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The  Military  Operations  Research  Society 

The  purpose  of  the  Military  Operations  Research  Society  (MORS)  is  to  enhance  the  quality  and 
effectiveness  of  classified  and  unclassified  Military  Operations  Research.  To  accomplish  this 
purpose,  the  Society  provides  media  for  professional  exchange  and  peer  criticism  among 
students,  theoreticians,  practitioners,  and  users  of  Military  Operations  Research.  These  media 
consist  primarily  of  the  traditional  annual  MORS  Symposia  (classified),  their  published 
proceedings,  special  mini-symposia,  workshops,  colloquia  and  special  purpose  monographs.  The 
forum  provided  by  these  media  is  directed  to  display  the  state  of  the  art,  to  encourage  consistent 
professional  quality,  to  stimulate  communication  and  interaction  between  practitioners  and  users, 
and  to  foster  the  interest  and  development  of  students  of  operations  research.  In  performing  its 
function,  the  Military  Operations  Research  Society  does  not  make  or  advocate  official  policy  nor 
does  it  attempt  to  influence  the  formulation  of  policy.  Matters  discussed  or  statements  made 
during  the  course  of  its  Symposia  or  printed  in  its  publications  represent  the  positions  of  the 
individual  participants  and  authors  and  not  of  the  Society. 

The  Military  Operations  Research  Society  is  operated  by  a  Board  of  Directors  consisting  of  30 
members,  28  of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four  years.  The  persons 
nominated  for  this  election  are  normally  individuals  who  have  attained  recognition  and 
prominence  in  the  field  of  Military  Operations  Research  and  who  have  demonstrated  an  active 
interest  in  its  programs  and  activities.  The  remaining  two  members  of  the  Board  of  Directors  are 
the  immediate  Past  President  who  serves  by  right  and  the  Executive  Vice  President  who  serves  as 
a  consequence  of  his  position.  A  limited  number  of  Advisory  Directors  are  appointed  from  time 
to  tinie,  usually  for  a  one-year  term,  to  perform  some  particular  function.  Since  a  major  portion 
of  the  Society's  affairs  is  connected  with  classified  services  to  military  sponsors,  the  Society  does 
not  have  a  general  membership  in  the  sense  of  other  professional  societies.  The  members  of 
MORS  are  the  Directors,  persons  who  have  attended  a  MORS  meeting  within  the  past  three  years 
and  Fellows  of  the  Society  (FS)  who,  in  recognition  of  their  unique  contributions  to  the  Society, 
are  elected  by  the  Board  of  Directors  for  life. 

MORS  is  Sponsored*  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director,  Assessment  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Modeling,  Simulation  and  Analysis,  Deputy  Chief  of  Staff,  Air  and  Space, 
Plans  and  Operations,  Headquarters,  US  Air  Force 

•  The  Commanding  General,  Marine  Corps  Combat  Development  Command 

•  The  Director  of  Force  Stracture,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 


As  of  September  1994 
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PREFACE 

The  Simulation  Validation  (SIMVAL)  series  of  Workshops  and  Mini-Symposia  started  in 
October  1990  in  Albuquerque  NM.  I  attended  the  original  Workshop  and  SIMVAL  11  in  1992. 
As  a  participant  in  SIMVAL  '94  and  a  Co-Chair  of  Working  Group  (WG)  4A  on  Advanced 
Distributed  Simulation  (ADS),  I  feel  these  discussions  on  Verification,  Validation  and 
Accreditation  (VV&A)  have  served  a  fine  purpose  in  alerting  the  Military  Operations  Research 
Community  to  a  need  to  regularize  discussions  on  these  topics.  The  topics  of  W&A  have 
achieved  recognition  in  the  defense  community  with  DoD,  the  Joint  Staff  and  the  Services 
issuing  directives  and  instructions  on  how  to  make  VV&A  happen  on  a  regular  basis.  This  is  a 
tangible  sign  that  MORS  has  succeeded  very  well. 

The  Executive  Summary  was  written  originally  by  Jim  Sikora,  FS  and  Marion  Williams,  FS  as  an 
article  for  the  December  1994  issue  of  PHALANX  and  is  reproduced  in  its  entirety  here.  These 
Proceedings  expand  on  that  material  by  presenting  in  detail  the  summaries  of  the  several  working 
groups. 

hi  September  1994,  at  the  time  of  the  SIMVAL  ’94,  DIS  was  the  most  notable  example  of  ADS 
and  WG  4A  and  4B  devoted  considerable  attention  to  it.  DIS  has  been  superceded  by  the  High 
Level  Architecture  (HLA)  and  currently  is  in  widespread  use  at  the  time  of  this  publication. 

Julian  Palmore 
Editor 
April.  1997 
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EXECUTIVE  SUMMARY 

SIMVAL  Co-Chairs;  Dr.  Marion  L.  Williams,  FS  and  Mr.  James  J.  Sikora,  FS 


This  publication  is  the  proceedings  of  the  Military 
Operations  Research  Society  (MORS)  Simulation 
Validation  (SIMVAL)  Workshop  held  28-30 
September  1994  in  Albuquerque,  New  Mexico 
(SIMVAL  ‘94).  SIMVAL  '94  was  hosted  by 
BDM  International  and  the  Air  Force  Operational 
Test  and  Evaluation  Center.  The  Workshop  was 
one  of  the  SIMVAL  series  on  Modeling  and 
Simulation  (M&S)  Verification,  Validation  and 
Accreditation  (W&A).  It  was  specifically 
structured  to  address  the  theme  “How  Much  V&V 
(Verification  and  Validation)  is  Enough?”  These 
Proceedings  contain  the  findings  of  the  working 
groups  as  well  as  a  summary  of  those  findings. 

Chapter  1,  "Introduction,"  provides  the  purpose  of 
the  Workshop,  the  four  key  questions  which  were 
discussed  and  answered  in  order  to  address  the 
theme  and  the  overall  approach  to  the  Workshop 
organization  and  procedures.  Chapter  2, 
"Background  of  the  SIMVAL  Series,"  provides  a 
brief  history  of  the  SIMVAL  series  and  a 
description  of  its  previous  work.  Working  group 
(WG)  summaries  are  provided  in  Chapters  3 
through  7.  Each  chapter  contains  a  summary  of 
the  findings  of  a  WG  relative  to  the  four  key 
questions.  The  WG  summaries  were  taken  from 
annotated  briefings.  The  briefings  were 
developed  by  the  working  group  co-chairs  and 
presented  during  the  last  session  of  the  workshop. 
Chapter  8,  "Workshop  Summary,"  integrates  the 
findings  of  all  of  the  working  groups  into  a  single 
set  of  responses  to  the  four  key  questions. 
Chapter  9,  "Where  Do  We  Go  From  Here?" 
provides  a  summary  of  the  recommendations  from 
the  SIMVAL  Senior  Advisory  Group  (SAG) 
meeting  that  was  held  immediately  after  the 
SIMVAL  '94  Workshop  concluded. 

It  is  a  basic  premise  of  the  SIMVAL  series  that  its 
findings  represent  a  consensus  of  the  military  and 
the  military  support  community.  These 
Workshop  findings  represent  such  a  consensus.  It 
is  built  not  only  on  the  work  of  the  attendees,  but 
on  the  efforts  of  several  hundred  other 
participants  in  previous  SIMVAL  activities 


during  the  past  four  years.  We  owe  a  great  deal  of 
gratitude  to  all  these  people  for  their  hard  work 
and  dedication.  Their  elfort  will  not  have  been  in 
vain  —  if  the  community  will  use  their  findings  as 
a  basis  for  implementing  W&A  policies  and 
procedures  and  in  specific  VV&A  applications. 

1.  SIMVAL  Background 

The  first  effort,  a  Mini-Symposium  held  in 
Albuquerque,  New  Mexico  15-18  October  1990, 
was  hosted  by  the  Air  Force  Operational  Test  and 
Evaluation  Center  and  BDM  International.  It 
provided  a  foram  for  discussing  ongoing  activities 
in  model  validation  and  served  as  a  basis  for 
planning  future  efforts. 

Objectives  of  the  Mini-Symposium  were  to: 

•  Review  current  efforts  in  simulation 
validation. 

•  Support  technical  interchange  on  simulation 
validation. 

•  Agree  on  a  consistent  set  of  definitions  for 
terms  such  as  "validation,"  "verification," 
"accreditation,"  etc. 

•  Develop  a  plan  for  future  efforts  to  address 
issues  of  simulation  validation. 

At  this  Mini-Symposium,  a  set  of  definitions  was 
adopted,  and  later  approved  by  the  MORS 
Sponsors.  These  definitions  have  been  accepted 
by  the  Services  and  OSD,  and  have  provided  the 
basis  for  much  of  the  VV&A  policies  now  in  use 
throughout  DoD. 

The  definitions  adopted  at  that  conference  were: 

•  Verification.  The  process  of  determining  that 
a  model  implementation  accurately  represents 
the  developer’s  conceptual  description  and 
specifications. 
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•  Validation.  The  process  of  determining  the 
degree  to  which  a  model  is  an  accurate 
representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the  model. 

•  Accreditation.  An  official  determination  that 
a  model  is  acceptable  for  a  specific  purpose. 

At  subsequent  Workshops,  the  elements  of 

VV&A  were  defined.  These  included  the 

following: 

•  Code  Verification.  This  consists  of  a  rigorous 
audit  of  the  code  to  ensure  proper 
implementation,  accomplished  by  both  the 
developer  and  an  independent  IV&V  agent. 

•  Comparison  to  Intelligence  Data. 
Comparison  of  the  model  output  with 
collected  intelligence  data  or  intelligence 
analyses. 

•  Comparison  with  Developmental  Data. 
Comparison  of  common  measures  in  the 
model  to  those  obtained  in  a  test  of  the  system 
under  laboratory  or  controlled  test  conditions. 
(This  provides  a  way  of  understanding  or 
calibrating  the  model  results  where 
appropriate) 

•  Comparison  with  Operational  Data. 
Comparison  of  the  model  output  with  results 
of  field  exercises,  operational  tests  or  actual 
system  or  combat  results. 

•  Configuration  Management.  A  discipline 
applying  technical  and  administrative 
oversight  and  control  to  identify  and 
document  the  functional  requirements  and 
capabilities  of  a  model,  control  changes  to 
those  capabilities  and  document  and  report 
the  changes.  Configuration  management 
includes  ensuring  the  detailed  design  and  the 
computer  source  code  of  the  model  are 
properly  documented.  This  should  be 
accomplished  in  conjunction  with  the  model 
developer. 


•  Data  Verification.  Review  of  the  sources  of 
data  and  consistency  of  definition  of  how  data 
were  collected  (conditions,  limitations,  etc.) 
along  with  a  definition  of  how  the  data  are 
used  in  the  model. 

•  Data  Validation.  Comparison  of  model  input 
data  to  the  corresponding  known  real  world  or 
best  estimate  values.  This  is  typically  done 
by  the  model  user. 

•  Documentation.  Analyst's  manual,  user's 
guide,  programmer's  manual  etc.,  providing 
the  math,  program  structure,  assumptions  and 
algorithms  used.  This  also  includes 
documentation  of  procedures  and  results  of 
any  W&A  efforts. 

•  Face  Validation.  The  process  of  determining 
whether  a  model,  beised  on  performance, 
seems  reasonable  to  people  who  are 
knowledgeable  about  the  system  under  study. 
("Simulation  Modeling  and  Analysis,"  Law 
and  Keaton,  McGraw-Hill).  This  process 
applies  the  knowledge  and  understanding  of 
experts  in  the  field.  It  can  be  accomplished 
inexpensively  and  in  a  reasonably  short  time. 
It  is,  however,  subject  to  the  biases  and 
weaknesses  of  the  experts  involved. 

•  Independent  Review.  Examination  of  all  steps 
in  the  validation  process  by  an  agent 
independent  of  the  model  developer.  Subject 
to  the  competence  and  independence/ 
objectivity  of  the  reviewer. 

•  Logical  Verification.  The  identification  of  a 
set  of  assumptions  and  interactions  for  which 
the  model  correctly  produces  intended  results. 
Logical  verification  determines  the 
appropriateness  of  the  model  for  a  particular 
application.  This  is  accomplished  by  the 
model  designer  or  the  developer's  IV&V 
agent. 
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•  Sensitivity  Analyses.  Using  the  full  range  or 
maximum,  minimum  and  high  probability 
values,  of  major  model  variables  in  model 
exercises  to  assure  correct,  corresponding 
changes  in  model  output. 

n.  SIMVAL  '94 

The  objective  at  SIMVAL  '94  was  to  address  the 
question:  "How  Much  V&V  is  Enough?"  We 
have  established  that  V&V  are  necessary,  and  that 
accreditation  is  the  decision  that  the  V&V  of  a 
particular  model  is  sufficient  for  a  specific 
application.  Now  how  much  V&V  is  necessary 
considering  such  factors  as  level  of  model,  type  of 
decision,  etc.? 

This  major  issue  was  addressed  through  four  key 
questions: 

1.  ACCREDITATION  TEMPLATE 

a.  Can  a  template  be  developed  to 
support  accreditation? 

b. '-  What  would  such  a  template  look  like? 

c.  How  should  we  go  about  developing 
such  a  template? 

2.  V&V  STATUS 

a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

h.  Are  there  “units”  of  V &V? 

3.  ACCREDITATION  FACTORS 

What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

4.  LEGACY  M&S 

a.  What  is  a  legacy  model? 

b.  Does  a  legacy  M&S  acquire  any  V & V 
credibility  by  its  use  and  acceptance? 


c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

Four  SIMVAL  '94  working  groups  were 
established  to  address  the  issue  and  four  key 
questions.  Each  WG  represented  a  different  type 
of  models:  campaign  level,  mission  level, 
engagement  level  and  Distributed  Interactive 
Simulation  (2  WGs). 

1.  a.  Can  a  template  he  developed  to  support 
accreditation? 

All  working  groups  agreed  that  an  accreditation 
template  could  be  developed.  They  agreed  that 
one  template  was  not  possible;  there  should  be  a 
family  of  templates  based  on  M&S  level, 
application  type  and  use.  A  composite  of  the 
working  groups’  conclusions  include  the 
following  in  the  template: 

A.  Application  Description 

1 .  Purpose  of  application 

2.  Objectives 

3.  Specific  issues/concems 

4.  Application  risk/importance 

5.  Time  required  for  accreditation 

6.  Resource  constraints 

B.  Accreditation  Basics 

1 .  M&S  description  (Title,  M&S 
scope/overview,  version) 

2.  M&S  proponent  (reputation/experience) 

3.  Proposed  use  in  decision  process 
(integration  with  other  methods/data) 

4.  Key  functional  representations 

5.  V&V  status  (including  credentials  of 
V&V  agents) 

6.  Sensitivity  analyses  performed 

7.  Known  limitation/constraints 

8.  Configuration  management/control 

9.  Documentation  available 

10.  Data 

a.  Sources 

b.  Verification  status 

c.  Validation  status 

d.  Certification 

e.  Uncertainty 
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11.  Affordability  considerations. 

12.  Availability  of  M&S  and  trained, 
experienced  users 

13.  Risk  reduction  actions 

14.  Alternatives  considered 

15.  Historical  use,  acceptance,  known 
applications  (including  any  DIS  use) 

16.  I*revious  accreditation  history  (including 
any  DIS  use) 

C.  Assessment  Approach 

1 .  Acceptance  criteria  (by  key  functional 
representation) 

2.  Priority  sequence  of  key  functions  for 
V&V  (useful  if  under  a  time  or  resource 
constraints) 

3.  Methodology  for  meeting  criteria 

a.  Verification  plan 

b.  Validation  plan 

.  c.  DIS  considerations  (if  a  distributed 
interactive  simulation) 

d.  Impact  on  application  of  M&S 
functions  not  verified 

e.  Impact  on  application  of  M&S 
functions  not  validated 

D.  Recommendations/Options 

1.  Actions  taken  (if  any) 

2.  Caveats,  by  functional  representation 

E.  Decision 

The  accreditation  decision  maker  should  consider 
all  of  these  factors  before  accepting  an  M&S  or 
its  results,  but  the  degree  to  which  they  are 
considered  is  a  function  of  the  application 
risk/importance  as  well  as  resource  and  time 
constraints.  The  development  of  guidelines  for 
determining  the  degree,  or  how  much  is  enough, 
is  addressed  in  Section  in  below. 

The  group  agreed  that  were  no  quantifiable  units 
for  V&V.  They  also  said  that  if  any  such  units 
were  developed,  they  would  likely  be  misused. 


2.  a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

Another  issue  addressed  at  the  workshop  was  a 
methodology  for  presenting  the  V&V  status  of  a 
M&S  application.  The  working  groups  generally 
agreed  that  verification  and  validation  status 
should  be  reported  separately.  The  status  would 
be  conveyed  by  the  following: 

A.  Strategic  Indicators 

1.  Age/maturity 

2.  Level  of  configuration  management 

3.  Size  of  user  group 

4.  Major  versions/pace  of  change 

5.  Number  of  applications/studies  supported 

6.  Data  sources 

7.  Functional  adaptability 

B.  Verification  Status 

1 .  Key  functions/elements/interactions 

2.  For  each  key  function/element/interaction 

a.  Verification  method  used 

b.  M&S  version  to  which  applied,  with 
linkage  to  current  version 

c.  Date  of  application 

d.  Verification  agent 

e.  Verification  report  reference 

f.  Verification  results  (quantitative 
statements) 

g.  Verification  conclusions  (quantitative 
statements) 

C.  Validation  Status 

1 .  Key  functions/elements/interactions 

2.  For  each  key 
fiinctions/element/interaction 

a.  Validation  method  used 

b.  M&S  version  to  which  applied,  with 
linkage  to  current  version 

c.  Date  of  application 

d.  Validation  agent 

e.  Validation  report  reference 

f.  Validation  results  (quantitative 
statements) 

g.  Validation  conclusions  (quantitative 
statements) 
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D.  Limitations/Risks 

4.  What  Are  Accreditation  Factors?  The 
factors  are  shown  in  the  accreditation 
template  of  question  1. 

5.  How  do  we  W&A  a  "legacy"  M&S?  As 
an  additional  topic,  the  working  groups 
addressed  the  issue  of  "legacy"  M&S. 
Although  there  was  no  definition  of  a  legacy 
model,  most  agreed  that  a  legacy  M&S  was 
one  with  a  long  history  of  use  with  moderate 
or  high  levels  of  V&V  and  broad  acceptance 
in  the  community.  The  working  groups 
basically  agreed  that  legacy  M&S  accrued 
credibility  by  use  over  time,  but  should  go 
through  the  same  processes  for  W&A  as  any 
other  M&S.  It  should  have  a  larger  data  base 
of  historical  use  and  W&A  from  which  to 
draw,  thereby  making  an  accreditation 
decision  easier  or  quicker. 


HI.  SIMVAL  Guidance  on  SIMVAL  *94 
Results 

At  the  SIMVAL  SAG  meeting  held  following 
SIMVAL  '94,  it  was  agreed  to  use  the  output  of 
the  Workshop  to  draft  a  set  of  VV&A  templates. 
The  result  will  be  a  set  of  guidelines  for 
application  and  model  types.  These  guidelines 
will  be  put  on  an  electronic  bulletin  board  to  be 
developed  by  MORS.  This  will  allow  individuals 
to  comment  on,  and  make  changes  to,  the 
guidelines.  At  a  time  when  there  seems  to  be 
general  consensus,  these  guidelines  will  be 
published.  It  was  generally  agreed  that  no 
additional  SIMVAL  Workshops  should  be 
planned  until  the  community  has  had  time  to  use 
VV&A  guidelines  developed  by  the  SIMVAL 
series  in  case  studies.  At  some  future  date,  a 
workshop  to  discuss  these  case  studies  would  be 
beneficial. 
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CHAPTER  1 
INTRODUCTION 


A.  Purpose  of  Workshop 

The  SIMVAL  '94  Workshop  was  one  in  a  series 
of  MORS  activities  focused  on  the  area  of  Model 
and  Simulation  (M&S)  Verification,  Validation 
and  Accreditation  (VV&A).  The  overall  purpose 
of  all  SIMVAL  series  activities  is  to  bring 
interested  members  of  the  MORTM&S  community 
together  to  discuss  current  VV&A  problems  and 
issues,  and  to  develop  by  consensus,  solutions, 
approaches  and  recommended  practices  to  address 
these  problems  and  issues. 

The  purpose  of  the  SIMVAL  '94  Workshop  was 
to  address  the  theme,  "How  much  V&V  is 
enough?"  The  "enough"  in  the  theme  implies  for 
an  accreditation  decision  to  accept  a  M&S  or  its 
results.  The  theme  was  addressed  through 
consideration  of  four  key  questions. 

B.  The  Key  Questions 

Four  key  questions  which  consider  different 
aspects  of  the  theme  were  the  basis  of  Workshop 
discussion.  The  questions  are; 

1.  ACCREDITATION  TEMPLATE 

a.  Can  a  template  be  developed  to 
support  accreditation? 

b.  What  would  such  a  template  look  like? 

c.  How  should  we  go  about  developing 
such  a  template? 

2.  V&V  STATUS 

a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

b.  Are  there  “units”  of  V &V? 

3.  ACCREDITATION  FACTORS 


What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

4.  LEGACY  M&S 

a.  What  is  a  legacy  model? 

b.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

C.  The  Workshop  Approach 

The  approach  to  the  Workshop  was  to: 

•  Set  the  context  of  discussions  by  reviewing 
the  history/previous  findings  of  the  SIMVAL 
series.  This  was  accomplished  by  sending  out 
a  section  of  the  previous  SIMVAL  Workshop 
proceedings  as  read-  ahead  material  and  by  a 
short  briefing  to  start  the  SIMVAL  ‘94 
Workshop. 

•  Give  an  overview  of  the  current  Marine 
Corps,  OSD  (PA&E)  and  the  government 
Accounting  Office  (GAO). 

•  Break  up  into  small  working  groups  for 
discussion. 

•  Discuss  approaches/solutions  to  the  four  key 
questions. 

•  Summarize  the  findings  of  each  working 
group  and  present  the  summary  to  all 
attendees. 

The  Workshop  attendees  were  organized  into  four 
major  groups  based  on  types  of  M&S.  The  four 
groups  were: 

1 .  Campaign/Theater  Level 

2.  Mission  Level 


7 


3.  Engagement  Level 

4.  Advanced  Distributed  Simulation(ADS)/ 
Distributed  Interactive  Simulation  (DIS) 

Attendees  were  assigned  to  working  groups  based 
on  individual  preferences  given  on  attendance 
application  forms.  Because  of  the  large  interest  in 
Engagement  Level  and  Advanced  Distributed 
Simulation,  these  working  groups  were  divided 
into  two  sections  each  (3A,  3B,  4A  and  4B).  The 
working  groups  discussed  the  four  key  questions 
from  the  perspective  of  their  assigned  type  of 
M&S.  Each  WG  or  section  was  led  by  two  co¬ 
chairs.  The  co-chairs  are  listed  below. 

WGl  CAMPAIGN/THEATER  LEVEL 
MODELS/SIMULATIONS 

Lt  Col  Robert  Sheldon,  USAF 
Mr.  Howard  Whitley 

WG2  MISSION  LEVEL 

Ms.  Cheryl  Black 
Mr.  John  Riente 

WG3A/B  ENGAGEMENT  LEVEL  M&S 

Mr.  Robert  Hartling  (WG3A) 

Mr.  Mark  Ralston  (WG3A) 

Dr.  Paul  Muessig  (WG3B) 

Mr.  Michael  Borowski  (WG3B) 

WG4A  (ADS/DIS)  ADVANCED 
DISTRIBUTED 
SIMULATION/DISTRIBUTED 
INTERACTIVE  SIMULATION 

Dr.  Julian  Palmore 
Mr.  Robert  Eberth 

WG4B  (ADS/DIS)  ADVANCED 
DISTRIBUTED 
SIMULATION/DISTRIBUTED 
INTERACTIVE  SIMULATION 

Mr.  William  Dunn 
Mr.  Rob  Wright 


After  the  working  groups  met  for  the  first  time, 
the  two  sections  of  group  3  (3A  and  3B)  decided 
they  would  operate  as  a  single  working  group. 
Some  of  the  working  groups  began  their 
discussions  with  case  study  briefings  in  their  area. 
The  WG  co-chairs  led  their  assigned  attendees 
through  consideration  of  each  of  the  four  key 
questions.  At  the  end  of  the  discussion  period, 
the  findings  for  each  question  were  summarized 
and  developed  into  a  WG  summary  briefing. 
Each  working  group  presented  their  individual 
summary  briefing  at  the  end  of  the  Workshop. 
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CHAPTER  2 

BACKGROUND  OF  THE  SIMVAL  SERIES 


MORS  has  sponsored  a  series  of  activities  on 
"Simulation  Validation."  (SIMVAL)  as  shown  in 
Table  2-1.  The  first  activity,  a  mini-symposium 
held  in  Albuquerque,  New  Mexico,  October  IS¬ 
IS,  1990,  was  hosted  by  the  Air  Force 
Operational  Test  and  Evaluation  Center  and  BDM 
International  Inc.  The  Mini-Symposium  provided 
a  forum  for  general  discussion  of  the  broad  topic 
of  simulation  VV&A  and  served  as  a  basis  for 
planning  future  efforts. 

Objectives  of  the  Mini-Symposium  were  to: 


•  Develop  consensus  on  a  consistent  set  of 
definitions  for  terms  such  as  "verification," 
"validation"  and  "accreditation, "etc,. 

•  Develop  a  plan  for  future  efforts  to  address 
issues  of  simulation  validation. 

The  Mini-Symposium  was  divided  into  five  major 
sessions:  Requirements  Analysis,  System  Design, 
Operational  Test  and  Evaluation,  Operations 
Support  and  Tactics  Development  and  Training. 


TABLE  2-1  SIMULATION  VALIDATION  SERIES  ACTIVITIES 


13  February  1990 

SAG  Meeting  1  -  Planning 

14  June  1990 

58th  MORSS  -  Preliminary  community  discussion  on  series 
(WG  28) 

15-18  October  1990 

MINI-SYMPOSIUM  I 

16  October  1990 

SAG  Meeting  2  -  Review/discuss  definitions/roadmap 

18  October  1990 

SAG  Meeting  3  -  Establish  definitions  (V,V  &  A),  update 
roadmap 

12-13  December  1990 

AD  HOC  Working  Group  I  -  Validation  methodology 

7  February  1991 

SAG  Meeting  4  -  Review/discuss  accreditation,  update 
roadmap 

20  March  1991 

SAG  Meeting  5  -  Review/discuss  VV  &  A 

11-13  June  1991 

59th  MORSS  -  Summary  of  findings/status  report 

17  September  1991 

SAG  Meeting  6  -  Develop  monograph  approach/workshop 
structure 

31  March  -  2  April 

SIMVAL  II  WORKSHOP  -  Review  methods,  develop  basis  for 

1992 

monograph 

23-25  Jun3  1992 

60th  MORSS  -  Summary  of  fmdings/status  report  in  General 
Session 

3  March  1993 

SAG  Meeting  7  -  Review  monograph  approach/update 
roadmap 

18  November  1993 

SAG  Meeting  8  -  Develop  approach  to  SIMVAL  94 
WORKSHOP 

28-30  September  1994 

SIMVAL  '94  WORKSHOP 

30  September  94 

SAG  Meeting  9  -  Determine  future  directions 

•  Review  current  efforts  in  simulation 
validation. 

•  Support  technical  interchange  on  simulation 
validation. 


Papers  for  these  sessions  included  case  histories, 
methodologies,  lessons  learned  and  status  of 
current  simulation  validation  efforts. 

A  SAG,  composed  of  senior  analysts  representing 
a  breadth  of  simulation  experience,  was  formed  to 
provide  guidance  in  planning  the  Workshop 
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series,  to  assist  in  developing  a  consistent  set  of 
definitions,  and  to  develop  a  roadmap  of  activities 
necessary  to  arrive  at  a  consensus  on  a  model 
validation  process.  The  SAG  membership  is 
shown  in  Table  2-2.  The  goal  of  the  SAG  was  to 
arrive  at  a  consistent  set  of  definitions  for 
simulation  verification,  validation  and 
accreditation  which  would  be  agreeable  to  all 
DoD  components,  thus  resolving  the  problems 
caused  by  the  current  use  of  different  definitions. 

The  SAG  recommended  a  subsequent  meeting  to 
provide  a  better  description  of  the  validation 
methodologies.  To  accomplish  this,  an  ad  hoc 
WG  meeting  was  held  at  the  MITRE  Corporation 
on  12-13  December  1990,  with  DoD  component 
and  industry  representatives.  The  purpose  of  the 
meeting  was  to  attempt  to  define  elements  of  a 
validation  process.  Experts  in  five  different  types 
of  application  areas  were  invited;  force  planning 
and  operations,  acquisition,  test  and  evaluation, 
training  and  deployment,  mobilization  and 
sustainability. 

SIMVAL  n,  another  Workshop  of  the  SIMVAL 
series- was  held  31  March  -  2  April  1992,  at  the 
Institute  for  Defense  Analyses  (EDA).  At  this 
Workshop,  model  VV&A  case  studies  were 
discussed,  and  examples  were  mapped  into  the 
VV&A  elements  defined  at  previous  meetings. 
The  concept  was  to  use  the  most  pertinent 
portions  of  the  case  studies  as  examples  of 
specific  elements  of  W&A. 


TABLE  2-2  SENIOR  ADVISORY  GROUP 
MEMBERS 


Dr.  Marion  Williams,  FS, 
(Co-chair) 

AFOTEC 

Jim  Sikora,  FS,  (Co-chair) 

BDM 

International 

Dr.  David  Anderson 

USAFSAA 

Dr.  Paul  Davis 

RAND 

Dr.  Hank  Dubin 

USAOPTEC 

Jim  Duff 

USN  OPTEVFOR 

Bill  Dunn 

USA  MISMA 

Christine  Fossett 

GAO 

Dr.  Saul  Gass 

Uni  V  of  MD 

Dr.  Dale  Henderson 

LANL 

Dr.  Ron  Hofer 

USA  STRICOM 

Dr.  Mort  Metersky 

USNNADC 

Allen  Murashige 

USAF SAA 

Jim  O’Bryon 

OSD. 

T&E/L&MP 

Dr.  Dale  Pace 

JHU/APL 

Nelson  Pacheco 

IDA 

Dr.  Julian  Palmore 

Univ  ofIL 

John  Riente 

DSMC 

Kathleen  Ruemmele 

BMDO 

Dr.  Pat  Sanders 

OSDPA&E 

Dr.  Ernest  Seglie 

OSDDOT&E 

Dennis  Shea 

CNA 

Clay  Thomas,  FS 

USAF  SAA 

Gene  Visco,  FS 

USA  MISMA 

Mark  Zabek 

IDA 
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CHAPTER  3 
WORKING  GROUP  1 

CAMPAIGNH'HEATER  LEVEL  MODELS/SIMULATIONS 

Co-chairs:  Lt  Col  Robert  Sheldon  and  Mr.  Howard  Whitley 


This  is  a  report  of  the  activities,  discussions  and 
results  of  WG  1,  SIMVAL  '94.  Answers  to  the 
following  questions  were  addressed. 

1.  ACCREDITATION  TEMPLATE 

a.  Can  a  template  be  developed  to 
support  accreditation? 

b.  What  would  such  a  template  look 
like? 

c.  How  should  we  go  about  developing 
such  a  template? 

2.  V&V  STATUS 

a.  How  do  we  present  tihe  V&V  status  of 
aM&S? 

b.  Are  there  “units”  of  V&V? 

3.  ACCREDITATION  FACTORS 

What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

4.  LEGACY  M&S 

a.  What  is  a  legacy  model? 

b.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

The  WG  responded  to  the  questions  relating  to 
the  possibility  of  developing  templates  to 
support  accreditation  decisions  as  follows.  The 
group  agreed  that  such  templates  can  be 
developed.  The  template  must  require 


consideration  of  the  scenario  in  the  application, 
the  quality  of  data  available  to  describe  such 
parameters  as  lethality,  vulnerability,  etc.,  the 
experience  of  analysts  using  the  model  or 
simulation  in  the  application  as  well  as  detailed 
information  on  the  M/S.  Important  issues  that 
should  be  addressed  are  acceptability  criteria, 
methodology  for  meeting  criteria,  results  and 
indicated  actions  and  accreditation  degrees: 
provisional  (with  caveats,  by  functional 
representation)  and  final  (credibility  of  result). 
This  suggests  the  possibility  that  often  full 
accreditation  may  not  be  recommended  until 
either  the  study  is  well  underway  or  completed. 
A  provisional  accreditation,  with  caveats 
regarding  certain  functional  representations, 
might  be  granted  pending  either  receipt  of  more 
complete  information  or  completion  of  the 
analysis.  In  the  latter  instance  the  credibility  of 
results  might  result  in  a  final  accreditation 
decision.  While  the  WG  feels  it  has  listed  many 
of  the  most  important  M/S  specific  points  in 
putting  together  a  template,  there  are  probably 
other  important  items  that  must  be  included,  so 
the  best  template  would  evolve  over  time  and 
application  as  more  experience  in  VV&A  is 
achieved. 

2.  a.  How  do  we  present  the  V&V  status  of 
aM&S? 

b.  Are  there  “units”  of  V &V? 

Regarding  the  presentation  of  the  V&V  status  of 
a  M/S,  there  was  full  agreement  that  a  single 
number  would  not  suffice  —  that  it  could  not 
reflect  enough  information  to  be  useful  in  any 
decision.  The  group  felt  that  a  set  of  "strategic 
indicators"  should  be  developed,  that  a  list  of 
methods  used  in  verification  activities  as  well  as 
the  findings  would  best  describe  verification 
status.  That  the  validation  activities  carried  out 
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be  listed,  and  a  statement  of  known  M/S 
limitations  and  associated  risks  be  provided  for 
consideration  in  any  accreditation  decision. 
Note  that  these  are  mostly  qualitative 
assessments  and  statements. 

More  details  on  the  strategic  indicators,  or 
"pedigree,”  are  listed  below  in  Table  3-1. 
Although  it  is  not  yet  complete,  Table  3-1 
reflects  the  concern  that  the  complete 
background  be  open  for  review.  For  older  M/S, 
has  there  been  a  history  of 
review/improvement/testing  to  increase  one's 
confidence  in  its  credibility  and  acceptance? 
What  types  of  analyses  has  it  supported?  What 
is  the  user  community  consensus  regarding  its 
utility  and  credibility?  What  are  the  sources  of 
key  data  used?  What  are  the  major  functional 
representations?  Is  there  balance  across 
functional  representations? 

TABLE  3-1  STRATEGIC  INDICATORS 


Age/maturity 

Configuration  management 

Size  of  user  group 

Studies  supported 

Major  changes  and  pace  of  change 

Number  of  applications 

Community  consensus 

Data  sources 

Functional  representations 
Functional  balance  and  adaptability 
Systems  and  integration  assessment 
Characterization  of  outputs 
Model  timing  aspects 
Stochastic  vs  deterministic 


The  group  suggested  that  one  could  develop  a 
list  of  functions  common  to  most,  with  each 
representation  possibly  subdivided  one  or  more 
times  according  to  issues  being  addressed  in  an 
application. 

The  WG's  consensus  on  units  of  V&V  was  "No, 
no,  a  thousand  times  no!"  One  notes  that  the 
status  of  a  M/S  is  best  represented  by  its 
pedigree  and  qualitative  statements  regarding  its 


V&V  history,  by  tabular  information  and  by 
"check-lists"  of  functionalities.  Though  one  can 
add  apples,  oranges,  grapes  and  other  odd-sized 
items  to  arrive  at  a  count  of  "things,"  to  try  to  do 
something  similar  in  this  respect  would  be 
hopeless. 

3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

The  WG  referred  back  to  the  template  for  the 
correct  inputs.  One  should  also  review  the  M/S 
"strategic  indicators." 

4.  a.  What  is  a  legacy  model? 

h.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

There  was  agreement  that  many 
campaign/theater  level  M/S  would  be  classed  as 
legacy  M/S  because  of  their  long  use,  high  to 
moderate  level  of  acceptability  and  low  level  of 
effort  in  the  V&V  area.  Because  of  this  long 
history  most  have  achieved  some  credibility  or 
face  validity.  Users  know  their  limitations,  and 
typically  most  problem  areas  are  known  within 
the  analysis  community.  The  group  felt  that  all 
our  V&V  efforts  on  these  would  be  less 
productive  than  systematically  addressing 
critical  functionalities  over  time,  to  include 
reverse  engineering  conceptual  models  against 
which  validation  assessments  can  be  made. 

WG  1  decided  that  additional  things  needed  to 
be  stated  regarding  V&V  for  campaign/theater 
level  models.  Because  of  the  problems  and 
challenges  of  validating  this  type  of  M/S,  some 
part  of  the  burden  of  V&V  must  be  shifted  to 
the  application.  "Reality  checks"  or 
examination  of  evolving  results  at  the  functional 
level  in  critical  representations  to  the  analysis  at 
hand  must  be  continuous.  The  need  to  fill  in 
modeling  "gaps"  revealed  through  their 
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application  in  new  areas  further  complicated  the 
parallel  process  suggested  above. 
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CHAPTER  4 
WORKING  GROUP  2 
MISSION  LEVEL 

Co-chairs:  Ms.  Cheryl  Black  and  Mr.  John  Riente 


1.  a.  Can  a  template  be  developed  to  support 
accreditation? 

The  group  concluded  that  a  general  set  of 
guidelines  can  be  developed  for  mission  level 
models  but  there  is  no  single  standard  for  applying 
the  template  to  all  accreditations.  For  example, 
accreditation  of  a  mission  level  model  to  support  a 
training  exercise  would  consider  different  factors 
and  may  assume  different  weights  for  specific 
factors  than  would  accreditation  for  the  same 
model  to  support  operational  planning. 

1.  b.  What  would  such  a  template  look  like? 

The  group  assumed  that  accreditation  would  be 
conducted  by  someone  (action  agent)  other  than 
the  decision  authority  (accreditation  official). 
The  action  agent  is  the  individual,  agency  or  ad 
hoc  committee  designated  to  conduct  the 
accreditation.  The  template  must  include  a 
charter  (Terms  of  Reference/Tasking  Document) 
issued  by  the  decision  authority  to  the  action 
agent.  The  charter  serves  as  a  formal  means  of 
communication  among  the  accreditation  authority, 
accreditation  action  agent  and  the  M/S  proponent. 
By  documenting  the  purpose  of  the  application, 
objectives  and  specific  concems/issues  for  the 
intended  use  of  the  model,  the  charter  establishes 
the  scope  and  focus  for  the  accreditation.  The  key 
measures  of  effectiveness,  objectives  of  the 
application,  features  of  the  model  of  high  interest 
and  expectations  of  the  decision  authority  would 
also  be  identified. 

The  accreditation  action  agent  prepares  the 
accreditation  plan.  The  plan  describes  the 
approach  to  be  taken  to  address  accreditation 
official  guidance,  identifies  responsibilities, 
resources  required  and  provides  a  milestone 
schedule.  The  factors  to  be  considered  in  the  plan 


are  identified  in  the  response  to  question  3. 

The  conduct  of  the  accreditation  assessment  is 
addressed  in  the  group's  response  to  question  2. 
The  WG  members  felt  that  the  assessment  should 
be  accomplished  at  two  key  decision  points; 

•  Early  in  the  study  planning  process  to  help 
make  model  selection  decision  and  to  identify 
actions  that  would  improve  model 
credibility/utility  for  the  application. 

•  At  the  end  of  the  application  to  assess  model 
modifications  made  during  the  course  of  the 
application. 

The  accreditation  assessment,  results  and 
recommendations  should  be  formally 
documented. 

1.  c.  How  should  we  go  about  developing 
such  a  template? 

The  WG  decided  that  there  is  no  "pass/fail" 
criteria  for  applying  the  template.  The 

accreditation  decision  must  consider  the 

following:  the  criticality  and  risks  of  the 
application  (e.g.,  battle  operational  plan  is 
different  than  research  investment  decision)  the 
scope  of  the  application  (e.g.,  battalion  level 
mission  or  Joint  Task  Force  operation)  and  the 
model  choices  (e.g.,  in  some  cases  only  one 
model  may  exist).  It  was  agreed  that  the 
guidelines  must  be  exceptionally  broad 

applications  and  environments.  Due  to  the 
exceptionally  broad  applications  of  M/S  the 
general  template  would  need  to  be  tailored  for 
applications  within  domains  (e.g.,  training,  RDA, 
operations,  etc.)  The  template  will  evolve  with 
use  and  its  credibility  will  grow  with  experience. 
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Doctrine/CONOPS . 


2.  a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

Verification.  The  group  concluded  that 
verification  of  mission  level  models  can  be  done. 
The  process  should  concentrate  on  model 
functions/objects  (and  their  interrelationships) 
which  are  critical  to  the  particular  application(s) 
of  the  model.  Documentation  is  vital  for  the 
verification  process,  especially  the  functional 
description  and  analysts  manuals  which  describe 
the  model  design,  algorithms,  assumptions,  etc.  If 
"good"  documentation  does  not  exist,  verification 
becomes  time  consuming  and  expensive.  The 
verification  process  should  include  scrutiny  of  the 
code,  not  just  review  of  the  documentation.  A 
verification  report  should  specifically  address 
those  functions/objects  which  are  critical  to  the 
proposed  use  of  the  model.  The  report  should 
identify  critical  functions  verified  and  those  not 
verified.  The  assessments  should  be  qualitative 
and  should  identify  the  risks  associated  with  not 
verifying  parts  of  the  model.  The  resources  and 
milestones  for  correcting  deficiencies  should  be 
identified. 

Validation.  The  group  determined  that  validation 
of  mission  level  models  is  challenging  and  may 
never  be  fully  accomplished.  This  is  attributed  to 
the  fact  that  it  is  difficult  to  define  "the  real 
world"  for  Joint  Task  Force  operations. 
Validation  of  mission  level  models  should 
concentrate  on  the  elements  and  interactions 
critical  to  the  proposed  application  of  the  model. 
Critical  elements/interactions  include: 

•  Consistency  with  physics  and  engineering 
level  representations. 

•  System  and  unit  performance. 

•  Environment  -  Natural  -  terrain,  weather, 
topography,  etc. 

•  Created  -  communications,  ECM,  ECCM, 
radar,  IR  transmissions. 

•  Command/Control/Intel. 


• 

•  Input  data  and  assumptions. 

A  validation  report  should  include  a  description 
of  the  proposed  application  of  the  model, 
definition  of  the  "real  world"  for  the  application, 
and  identification  (list)  of  the 
elements/interactions  critical  for  the  application. 
The  report  should  outline  historical  validation 
efforts  to  include  evolution  to  the  current  model 
version  and  applicability  to  the  current 
application.  An  assessment  of  the  "holes" 
identified  in  previous  validation  efforts  and  a 
proposed  methodology  for  filling  the  holes  in  the 
current  validation  should  be  included. 

The  group  recommended  that  the  community 
standardize  V&V  reporting  format.  This  would 
facilitate  subsequent  validation  efforts.  The  V&V 
documentation  should  be  archived  and  that  a 
central  source  for  information  regarding  the 
location  of  V&V  efforts  should  be  established. 

For  mission  level  models  Face  Validation  may  be 
particularly  useful.  Training  ranges,  exercises 
and  advanced  warfighting  experiments  are 
increasingly  being  used  in  the  context  of  DIS. 
The  community  should  take  advantage  of  these 
opportunities  to  validate  models  and  simulations. 
Analysts/agencies  tasked  with  model  V&V  should 
endeavor  to  make  better  use  of  data  from  man-in- 
the-loop  simulators  and  results  of  comparable 
models  which  have  been  validated  for  similar 
applications.  Models  that  have  been  used  in  the 
model-test-model  mode  or  compared  to  recent 
military  operations  are  especially  useful  in  this 
role. 

•  Minority  Opinion.  One  member  of  the  work 
group  felt  strongly  that  if  one  can't  understand 
the  phenomenology  of  a  "new"  system  or 
concept,  then  you  shouldn’t  use  digital  models 
to  study  it  and  can't  "validate"  a  model  of  it! 
The  majority  of  the  group  believed  that  in  this 
circumstance  a  model  could  be  useful  to 
improve  understanding  of  the  new  system 
concept. 
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2.  b.  Are  there  “units”  of  V&V? 

The  group  concluded  that  'units'  cannot  be  applied 
to  quantify  the  extent  of  V&V.  The  emphasis  of 
V&V  should  be  to  enhance  model  credibility. 
This  necessitates  focusing  on  the  model 
application.  The  accreditation  authority  should  be 
more  concerned  with  impacts,  implications  and 
risks  associated  with  using  the  model  and  its 
results  then  on  some  scale  of  V&V  accomplished. 
The  extent  of  V&V  needed  should  depend  on 
how  the  model  results  will  be  used  and  the 
resources/risk  for  the  problem  being  investigated. 

3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

As  discussed  in  response  to  question  Ic.  there  is 
no  pass/fail  criteria  that  can  be  used  to  determine 
acceptability.  Some  factors  to  be  considered 
include: 

•  V&V  Status.  Define  what  elements/functions 
have  been  V&V'd  and  which  have  not. 

•  Known  limitations/constraints.  Availability 
of  data,  functions  not  represented;  limits  on 
the  functions  represented;  hard  constraints  on 
geographic  area,  number  of  systems,  etc.; 
aggregation  level;  hardware  or  software 
limitations. 

•  Documentation.  Levels  of  documentation 
available  should  include  at  the  minimum 
functional  needs/description  (functional 
decomposition),  design  objectives/baseline 
designs,  user's  manual  and  configuration 
management  documentation.  It  should  follow 
DoD  guidelines  for  software  life  cycle 
management. 

•  Clarity.  Should  be  able  to  explain  results 
(cause  and  effect)  and  explain  the  results  to 
system/object  performance. 

•  Sensitivity  Analyses.  Model  results  behave  in 
the  expected  way  when  the  phenomenology  or 
process  is  understood.  Unexpected  results 


call  for  more  detailed  examination  to  gain 
appropriate  understanding. 

•  Configuration  Management.  Enforces  good 

V&V;  provides  audit  trail  for  modification 
and  different  versions  of  the  model.  Provides 
historical  documentation  of  the  uses  of  the 
model  and  formal  procedures  for 

documenting  why  certain  modifications  were 
made  and  an  assessment  of  the  success  of  the 
changes. 

•  Functional  Representation.  Identify  mission 
needs  and  describe  why  mission  level  model 
was  important  for  the  decision.  Explain  why 
mission  level  model  was  essential  for 
examining  the  decision  criteria  for  the 
application.  This  helps  assess  the  strengths 
and  limitations  in  the  context  of  the  use  of  the 
model. 

•  Risk  Reduction.  Explain  what  was  done  to 
minimize  known  shortcomings  of  the  model. 

•  Alternatives  Considered.  Explain  why  the 
model  was  selected. 

•  Credential  of  V&V  agent. 

•  Model  Proponent/Developer. 

•  Proposed  Use  in  the  Decision  Process. 

•  Historical  Use. 

•  Affordability,  Availability,  Trained/ 
Experienced  Users. 

•  Risks. 

•  Previous  Accreditations. 

4.  a.  What  is  a  legacy  model? 

A  legacy  model  is  one  that  is  continually 

improved  over  time,  is  configuration  managed  and 

has  "broad"  acceptance. 
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4.  b.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

Acceptance  and  use  does  contribute  to  a  model's 
credibility,  but  this  does  not  imply  that  the  model 
should  be  exempt  from  the  V&V  process. 

4.  c.  Do  we  V&V  a  legacy  M&S  any 
inherently  than  any  other  M&S? 

No.  However,  a  legacy  model's  credibility  will  be 
enhanced  with  successful  applications. 


18 


CHAPTER  5 
WORKING  GROUP  3 
ENGAGEMENT  LEVEL  M&S 

Co-chairs:  Mr.  Robert  Hartling  and  Mr.  Mark  Ralston  WG  3A 
Co-chairs:  Dr.  Paul  Muessig  and  LCDR  Michael  Borowski  WG  3B 


The  summary  of  the  results  of  the  Engagement 
Level  M&S  WG  at  the  MORS  SIMVAL  '94 
Workshop  is  reported  here.  Due  to  the  high 
level  of  interest  in  the  Engagement  Level  WG, 
two  subgroups  (3A  and  3B)  were  formed.  The 
Chair  of  WG  3A  was  Bob  Hartling  of  the 
Warfare  Requirements  and  Analysis  Division  of 
the  Chief  of  Naval  Operations;  the  Co-Chair 
was  Mark  Ralston  of  the  Army  Materiel 
Systems  Analysis  Activity  (AMSAA).  The 
Chair  of  WG  3B  was  Dr.  Paul  Muessig  of  the 
Naval  Air  Warfare  Center,  China  Lake;  the  Co- 
Chair  was  LCDR  Mike  Borowski  of  the  Navy’s 
Operational  Test  and  Evaluation  Force,  Norfolk. 
In  practice,  WGs  3A  and  3B  stayed  together, 
and  the  WG  results  were  compiled  into  this 
surrunary,  reflecting  the  consensus  of  about  40 
M&S  professionals. 

The  summary  is  divided  into  three  sections.  The 
first  section  describes  the  Engagement  Level 
WG  approach  to  answering  the  conference 
questions.  The  second  section  describes  the 
WG's  detailed  answers  to  these  questions 
developed  over  the  course  of  the  Workshop. 
The  last  section  summarizes  future  directions 
implied  by  the  workshop  results.  Many  of  these 
were  taken  up  for  action  at  a  SAG  meeting  held 
at  the  close  of  the  Workshop. 

The  WG  session  began  with  a  series  of  case 
studies  in  accreditation.  The  case  studies  were 
included  to  communicate  lessons  learned  from 
actual  experience,  and  to  set  the  stage  for 
detailed  discussions  of  the  workshop  questions. 
Mark  Ralston  discussed  AMSAA's  efforts  at 
accrediting  EVADE.  Van  Underwood  of 
JHU/APL  discussed  the  simulation  management 
process  required  to  accredit  a  Tomahawk  M&S. 
Paul  Muessig  reviewed  the  results  of  an 
accreditation  requirements  study  conducted  by 


the  SMART  (Susceptibility  Model  Assessment 
and  Range  Test)  Project. 

The  session  proceeded  with  a  logical  analysis  of 
the  conference  questions  given  by  Paul  Muessig. 
The  aim  was  to  delimit  the  range  of  possible 
answers  to  the  Workshop  questions  by  analyzing 
the  questions  in  themselves  implied  in  the 
ability  to  answer  them.  The  analysis 
demonstrated  that  the  Workshop  questions 
implied  the  necessity  of  a  set  of  standard  (i.e., 
well-defined  and  articulated)  V&V  process 
elements  and  a  standard  way  to  report  results. 
This  conclusion  resonated  with  both  the  case 
studies  and  the  accreditation  requirements  study 
conducted  by  SMART. 

The  detailed  discussions  focused  on  each 
conference  question  in  turn,  evaluating  the 
applicability  of  individual  lessons  learned  and 
on  developing  a  consensus  about  the  necessity 
of  standard  V&V  process  elements  and 
products.  Each  question  was  then  reviewed  in 
detail  near  the  end  of  the  Workshop,  and  the 
following  summaries  were  produced. 

1.  a.  Can  a  template  be  developed  to 
support  accreditation? 

Based  on  the  case  study  lessons  learned  and 
intense  discussion,  WGS  concluded  that  an 
accreditation  template  could  and  should  be 
developed.  All  agreed  that  differing 
requirements  for  template  content  based  on 
modeling  level  or  application  did  not  preclude 
the  possibility  of  developing  one. 
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1.  b.  What  would  such  a  template  look  like? 

WG3  concluded  that  an  accreditation  template 
would  most  likely  consist  of  a  core  set  of 
standardized  (i.e.,  well-defined,  articulated  and 
accepted)  V&V  process  elements  and  products. 
Elements  from  this  list  would  be  chosen  to 
support  individual  accreditation  efforts  based  on 
application  requirements  and  clearly  defined 
M&S  acceptance  criteria  based  on  these 
requirements. 

1.  c.  How  should  we  go  about  developing 
such  a  template? 

WG3  determined  that  the  best  way  to  develop  an 
accreditation  template  would  be  to  begin  with 
the  core  set  of  V&V  elements  developed  by 
SMART  (from  its  accreditation  requirements 
study)  and  build  from  there  based  on  application 
types,  modeling  level  and  any  other  factors 
determined  to  be  pertinent.  TTie  medium  for 
development  of  such  templates  would  be  small 
workshops  with  limited  objectives  sponsored  by 
MORS.  The  initial  templates  developed  would 
then  be  distributed  to  the  wider  community  for 
comment  and  consensus  development. 

One  very  important  issue  arose  during 
discussion  of  the  above  questions:  Should  a 
minimum  standard  of  V&V  be  specified  for  all 
M&S?  Put  another  way,  is  there  a  minimum 
amount  of  V&V  information  that  should  be 
developed  for  a  model  before  it  is  given  serious 
consideration  for  use  in  an  application?  The 
group  was  strongly  divided  on  this  issue.  The 
general  consensus  (about  25  of  40)  held  that 
minimum  V&V  standards  would  be 
counterproductive  since  any  minimum  standard 
would  quickly  become  the  de  facto  standard. 
("If  the  minimum  weren’t  good  enough,  it 
wouldn't  be  specified  as  the  minimum.")  A 
strong  minority  opinion  held,  however,  that  an 
amount  of  V&V  sufficient  to  characterize  a 
model  (analogous  to  nutrition  labels  on  food 
products)  would  represent  a  useful  minimum 
before  rational  use  of  a  model  could  be  justified. 


2.  a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

WG3  determined  that  the  answer  to  this 
question  depended  on  the  target  audience.  For 
an  accreditation  authority,  it  was  felt  that  reports 
of  progress  to  date  on  a  pre-approved 
accreditation  plan  would  suffice,  as  opposed  to 
detailed  summaries  of  technical  material. 
However,  to  the  wider  M&S  community,  it  was 
felt  that  a  summary  of  progress  and  results  for 
each  standard  V&V  element  of  the  template 
developed  under  question  Ic.  would  provide  an 
important  snapshot  of  a  model's  V&V  status  and 
history.  Although  no  one  suggested  that  this 
information  alone  was  sufficient  for  any 
accreditation,  it  was  a  consensus  among  the 
group  that  there  was  a  dire  need  for  a 
standardized  reporting  format  for  V&V  results 
for  each  element  of  the  template.  It  was  further 
determined  that  a  need  exists  for  a  database  of 
previous  VV&A  efforts  and  results  so  as  to 
minimize  duplication  of  effort  and  reduce  the 
costs  of  accreditation.  It  was  noted  that  the 
SMART  Project  has  developed  both  standard 
reporting  formats  for  many  V&V  elements  and 
an  Accreditation  Support  Database  that  will 
shortly  be  administered  through  SURVIAC  for 
survivability  M&S.  These  inputs  would  prove 
especially  useful  as  a  starting  point  in  template 
development  under  the  previous  question. 

2.  b.  Are  there  “units”  of  V&V? 

It  was  the  consensus  of  WG3  that  there  are  no 
units  for  V&V  per  se.  V&V  requirements  are 
application  dependent,  and  as  such  there  is  no 
“one  size  fits  all”  way  to  quantify  the  V&V 
status  of  a  model.  It  was  further  agreed  that  it 
would  be  misleading  to  develop  and  promulgate 
application  —  independent  V&V  “metrics”  for 
general  use.  However,  for  M&S  community  at 
large  it  was  felt  that  a  V&V  database  could 
provide  a  central  mechanism  for  the 
accumulation  of  objective  V&V  evidence  that 
could  support  accreditation  decisions. 
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3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

WG3  had  a  hard  time  with  this  question.  On  the 
surface,  it  appeared  either  redundant  with 
Question  2,  or  simply  too  ambiguously  stated  to 
answer  in  any  unambiguous  way.  The  group 
finally  decided  to  interpret  the  question  as  a 
means  of  defining  a  filtering  process  for  model 
selection.  In  this  context,  all  the  items  listed  in 
the  Workshop  handouts  for  this  question  (e.g., 
V&V  status,  usage  history,  cost  of  application, 
etc.)  were  deemed  to  be  of  value  in  model 
selection  prior  to  use  for  a  specific  application. 
The  group  determined  that  several  items  were 
missing  from  the  list  that  deserved  mention: 

•  Data  Availability.  "The  data  are  the  model," 
was  the  most  common  refrain.  If  a  model’s 
data  sets  are  sparse  or  undocumented,  it's 
utility  is  similarly  limited. 

•  Configuration  Management  (C/M).  If  a 
model  is  not  well  managed,  its  results  may 
not  be  trustworthy,  V&V  notwithstanding. 
C/M  maintains  the  "shelf  life"  of  V&V. 
Poor  C/M  practice  leads  to  needless 
squandering  and  spoilage  of  hard-won  V&V 
gains. 

•  Documentation.  A  poorly  documented 
model  cannot  be  verified  or  validated 
without  major  expense.  The  quality  of  the 
documentation  also  affects  the  cost  of  model 
use,  in  that  learning  curves  become  steeper 
and  training  expenses  rise. 

•  Applicability.  Some  characterization  of  the 
model's  application  domain  was  felt  to  be  an 
important  factor  in  model  selection.  "If  all 
you  have  is  a  hammer,  everything  starts  to 
look  like  a  nail."  The  frequency  to  improper 
use  of  models  could  be  reduced  with  a 
concise  statement  of  application  domain  and 
utility. 

•  Community  Acceptance.  Similar  to  the 
answer  at  the  first  bullet  of  question  4b, 
WG3  felt  that  community  acceptance 


implied  a  de  facto  "face  validation"  of  the 
model,  especially  if  it  is  also  well  managed. 

The  group  added  that  a  V&V  Library  capturing 
this  information  would  be  especially  useful  as 
an  aid  to  model  selection  for  particular 
applications.  It  was  noted  that  V&V  is 
currently  not  a  major  M&S  selection  criterion 
because  of  the  paucity  of  V&V  information  and 
V&V  information  sources.  A  V&V  Library 
would  thus  not  only  aid  accreditation  once  a 
model  had  been  selected  for  use,  but  would  also 
aid  model  selection  and  prioritization  of  V&V 
objectives  in  support  of  accreditation. 

4.  a.  What  is  a  legacy  model? 

There  were  many  opinions  on  what  factors  a 
minimum  definition  of  "legacy"  model  would 
include.  The  group  moved  on  to  answer  the 
following  questions,  and  discovered  that  none  of 
the  answers  depended  on  an  answer  to  this 
question.  We  concluded,  therefore,  that  since 
the  term  "legacy"  had  no  bearing  on  V&V 
requirements  or  process  elements,  it  was  not  a 
particularly  useful  descriptive  term.  Reference 
was  made  to  the  definition  of  "legacy"  in  API 
16-1001  (the  Air  Force  draft  M&S  management 
policy),  where  the  key  difference  between 
legacy  and  other  simulations  was  lack  of  coding 
and  documentation  standards.  It  was  observed, 
however,  that  neither  of  these  factors  had  a 
bearing  on  current  usage  or  acceptance  of  legacy 
M&S.  Again,  the  group  concluded  that  "legacy" 
was  not  a  term  useful  in  answering  the  central 
Workshop  question,  "How  much  V&V  is 
enough?" 

4.  b.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

WG3  determined  that  any  model  gains 
credibility  through  use  and  acceptance.  In  fact, 
it  was  noted  that  legacy  M&S  may  have  more 
credibility  in  this  regard  than  new  M&S 
developments,  despite  the  latter's  adherence  to 
coding  and  documentation  standards.  Dips  in 
the  acid  bath  of  reality  were  deemed  to  be  more 
important  in  assaying  M&S  than  mere 
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adherence  to  standards,  at  least  early  in  a 
model's  usage  history.  Again,  the  term  "legacy" 
added  no  new  V&V  requirements  or  rethinking 
of  basic  Workshop  issues. 

4.  c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

WG3  determined  that  there  would  be  no 
difference  in  a  notional  accreditation  template 
between  legacy  and  other  M&S.  Although  it 
was  agreed  that  verification  of  legacy  models 
did  involve  a  reverse  engineering  of  the 
conceptual  model  and  design  specification  that 
would  not  be  required  of  new  model 
developments,  validation  and  accreditation 
requirements  would  be  the  same.  We  viewed 
the  legacy  model  accreditation  vice  new  model 
accreditation  as  a  non-issue. 
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CHAPTER  6 
WORKING  GROUP  4A 

Advanced  Distributed  Simulation/Distributed  Interactive 

Simuiation  (ADS/DIS) 

Co-chairs:  Dr.  Julian  Palmore  and  Mr.  Robert  Eberth 


Due  to  the  large  number  of  participants  electing  to 
work  in  the  ADS/DIS  working  groups,  the  Group 
was  subdivided  into  two  independent  subgroups 
4Aand4B. 

The  Advanced  Distributed  Simulation/Distributed 
Interactive  Simulation  (ADS/DIS)  environment 
differs  from  other  environments  with  respect  to 
VV&A.  Our  WG  explored  W&A  from  this 
perspective. 

•  DIS  exercises  are  asynchronous;  there  is  no 
central  time  or  time  check. 

•  DIS  experiments  are  non-repeatable,  largely 
as  a  result  of  the  asynchronicity. 

A  DIS  exercise  or  experiment  consists  of  live, 
constructive  and  virtual  entities  and  cells, 
collections  of  entities,  networked  together  through 
the  use  of  highly  structured  standardized 
communication  protocols. 

To  be  DIS-compliant,  an  entity  or  cell  must 
adhere  only  to  the  applicable  Protocol  Data  Unit 
(PDU)  standards.  DIS-compliance  does  not  imply 
any  level  of  VV&A  has  been  performed. 

Moreover,  the  entities  in  an  exercise  may  be  at 
differing  levels  of  both  logical  and  presentational 
granularity,  and  at  differing  levels  of  fidelity  and 
validity. 

Thus,  W&A  must  be  addressed  at  two  levels  in 
the  DIS  environment; 

•  VV&A  of  the  entities  and  cells 

•  W&A  of  the  exercises 


we  approach  VV&A  of  legacy  models?"  We  used 
that  approach  and  reported  it  that  way  because  our 
deliberations  on  the  legacy  models  led  us  in 
particularly  productive  directions  with  respect  to 
the  remaining  questions. 

4.  a.  What  is  a  legacy  model? 

We  reached  a  consensus  definition  and  other 
definitions  of  interest. 

•  Consensus  Definition: 

»  Model  developed  in  the  past  that  is  still 
available  and  was  not  implemented  using 
current  standards. 

•  Minority  Definitions: 

»  DA  PAM  5-11:  Historically,  the  M&S  of 
choice  for  a  specific  application. 

»  OSD:  There  are  only  two  types:  Existing 
or  New. 

It  took  well  over  an  hour  to  come  to  a  consensus 
definition  for  a  "Legacy  Model."  While  this 
definition  is  the  consensus  of  the  WG,  both  of  the 
minority  definitions  had  numerous  strong 
adherents.  The  key  point  is  that  the  consensus 
definition  does  not  imply  any  favored  VV&A 
status  on  a  model  for  DIS  applications  simply 
because  it  may  be  "favored"  historically  for  one 
or  more  specific  applications. 

4.  b.  Does  a  legacy  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

Yes,  but  "credibility"  needs  to  be  viewed  in 
balance  with  all  other  accreditation  elements 
within  the  DIS  context. 


Our  WG  started  with  the  question,  "How  should 
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Certainly,  "use  and  acceptance"  imply  that  many 
persons,  including  a  number  of  domain  experts, 
have  looked  at  the  model  and  found  it  acceptable 
for  the  purpose  at  hand.  However,  such  use  and 
acceptance  is  only  a  limited  and  unstructured 
form  of  "face  validation,"  a  VV&A  technique  of 
problematic  rigor  and  power  to  begin  with.  Thus, 
"credibility  by  use  and  acceptance,"  in  itself, 
should  not  be  sufficient  for  accreditation.  This  is 
true  particularly  in  the  DIS  context,  where  each 
model  may  be  only  a  single  entity  or  cell  and  may 
be  interacting  with  other  DIS  compliant  entities  or 
cells  for  which  it  has  been  modified.  The  "other 
accreditation  elements"  will  be  seen  below. 

4.  c.  Do  we  V&V  a  legacy  M&S  any 
differently  than  any  other  M&S? 

For  all  models,  legacy  and  new,  an  accreditation 
plan  is  needed  that  addresses  the  model's  use  in  a 
specific  DIS  exercise. 

For  analytic  applications,  the  questions  to  be 
investigated  should  be  set  up  as  hypotheses  to  be 
tested  and  the  DIS  exercise  treated  as  experiments 
that  test  the  hypotheses.  This  is  presented  below 
as  an  Experimentalist  Framework  for  Distributed 
Interactive  Simulation. 

The  exercise  itself  should  be  accredited  through 
the  use  of  conceptual  models  and  metamodels  and 
techniques  to  measure  the  degree  of  compatibility 
and  consistency  between  entities. 

EXPERIMENTALIST  FRAMEWORK  FOR 
ADS/DIS  SIMULATION  EXERCISES" 

1.  State  the  problem. 

2.  Formulate  hypotheses  within  a  conceptual 
model. 

3.  Devise  experimental  techniques. 

4.  Examine  possible  outcomes  with  which  to 
evaluate  the  experimental  techniques. 

5.  Consider  possible  results  and  statistical 
procedures  to  validate  statistical  analyses. 


6.  Perform  the  experiments  as  DIS  exercises. 

7.  Draw  conclusions  by  analyzing  the 
experimental  results. 

8.  Evaluate  the  distributed  interactive 
simulation  experiment. 

While  the  question  is  cast  in  the  context  of 
Legacy  Models,  the  answer  is  cast  in  a  broader 
context.  This  is  the  heart  of  the  matter:  the  need 
to  introduce  scientific  method  into  the  VV&A 
process.  Here,  it's  tailored  to  the  DIS 
environment;  the  concept  is  applicable  across  the 
board. 

Breakthroughs  in  conceptual  thinking  and  in 
methodology  now  make  it  appear  realistic  to 
quantify  critical  aspects  of  "validity." 
Specifically,  the  "Experimentalist  Framework" 
shows  a  conceptualization  of  distributed 
interactive  simulation  exercises  that  resulted  from 
a  challenge  to  the  Military  Operations  Research 
community  by  Mr.  Walt  Hollis,  FS  the  DUSA  for 
Operations  Research,  to  find  ways  to  use  DIS 
exercises  for  analysis,  in  addition  to  training.  Its 
point  is  that  the  only  way  to  validly  employ  DIS 
exercises  for  analysis  is  to  first  state  the  analytical 
objectives  as  hypotheses  to  be  tested,  then  use  the 
exercises  as  carefully  structured  scientific  tests  of 
the  hypotheses.  The  same  concept  can  be  applied 
to  hypotheses  regarding  the  validity  of  individual 
entities  and  cells,  and  to  the  overall  exercise 
itself. 

'  J.  Palmore,  A  V&V  Framework  for  Analysis  in 
Distributed  Interactive  Simulations,  in  Object 
Oriented  Simulation  Conference  (OOS'94), 
Proceedings  of  the  1994  Western 
Multiconference,  Society  for  Computer 
Simulation,  January  1994,  pp  49-54. 

Other  work  reported  at  the  10th  DIS  Workshop 
included  the  potential  use  of  Fast  Fourier 
Transforms  to  quantify  the  consistency  among 
entities,  and  of  Colored  Petri  Nets,  which  can  be 
used  to  produce  quantitative  measures  of  merit,  to 
build  metamodels  of  DIS  exercises  prior  to  going 
to  the  time  and  expense  of  actually  putting  the 
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exercises  together. 

1.  a.  Can  a  template  be  developed  to  support 
accreditation? 

Yes.  We  found  the  first  part  of  the  question 
rather  straightforward  to  answer. 

1.  b.  What  would  such  a  template  look  like? 

The  template  should  be  based  on  the  DIS 
Workshop  VV«feA  Process  as  modified  by: 

•  DA  PAM  5-11,  plus  interfaces,  Tools,  use  of 
SIMTAX  and  interfaces  in  M&S  description. 
Verification  of  Architecture,  Data  V&V, 
Evaluation  Criteria  MOE's  and  M&S  Purpose. 

•  DMSO  IPL#2  Project  for  the  U.S.  Army 
V&V  of  Models  and  Simulations. 

We  believe  that,  at  least  for  DIS  accreditations, 
the  VV&A  Process  "template"  developed  by 
Robert  O.  Lewis  and  the  standing  DIS  W&A 
WG  provides  the  best  starting  point.  It  is  shown 
in  Figure  7-2.  The  DIS  Workshop  W&A 
Process,  however,  is  not  sufficient.  It  needs  to  be 
modified  to  include  additional  steps  and 
information  from  several  sources. 

•  Department  of  the  Army  Pamphlet  5-11 
contains  a  sample  Accreditation  Plan  format 
that  provides  for  other  extremely  useful  data 
to  lead  to  an  accreditation  decision. 

•  Similarly,  the  Simulation  Taxonomy 
(SIMTAX)  developed  by  MORS  and 
incorporated  in  the  J-8  catalog  of  models 
provides  additional  extremely  useful  data  on 
the  accreditation  template. 

1.  c.  How  should  we  go  about  developing  such 
a  template? 

We  should  modify  existing  prototypes  as 
appropriate. 

This  is  answered,  in  part,  by  a  template  based  on 


the  W&A  Process.  There  should  not  be  only  one 
template;  the  templates  should  be  tailored  to  the 
individual  accreditation  decision  environment. 
Data  elements  should  be  included  in  all  templates. 
In  particular,  the  data  captured  by  using 
SIMTAX  in  every  template  goes  a  long  way 
toward  enabling  useful  comparisons  of  model 
attributes. 

2.  a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

Presenting  the  V&V  status  is  done  by: 

•  Break  out  Verification  from  Validation; 
report  on  each  separately. 

•  For  Verification,  use  the  existing,  proven 
methods. 

•  For  Validation,  "by  another  mechanism." 

•  The  "other  mechanism." 

•  SIMVAL  H:  Validation:  "is  the  degree  to 
which  a  model  is  an  accurate  representation 
of  the  real  world." 

•  Establish  and  apply  MOEs  and  MOPs  from 
outset. 

•  Output  validation;  "extent  to  which  the 
output.” 

This  represents  the  significant  and  salient  features 
of  the  real  world,  comparison  of  real  world  data 
with  output  of  the  model,  coefficients  of 
correlation  and  confidence  levels. 

•  Structural  validation:  "extent  to  which  input 
data  and  the  conceptual  model  represent  the 
significant  and  salient  features,"  data  V&V, 
peer  review,  etc. 

We  suggest  reporting  the  status  of  the  two  Vs 
separately.  Verification  is  a  speciality  unto  itself. 
There  are  a  host  of  useful  tools  to  perform  it, 
starting  with  requirements  traceability  tools.  We 


25 


suggest  strongly  is  including  in  this  area  the 
results  of  the  peer  review  recommended 
previously.  As  discussed  in  the  SIMVAL  n 
report,  many  of  the  algorithm  checks,  reverse 
engineering  and  other  techniques  common  to  peer 
review  fall  generally  into  the  category  of  "logical 
verification." 

The  mechanism  we  suggest  for  reporting  the 
status  of  validation  is  SIMVAL  H,  specifically,  by 
developing  and  applying  statistical  data, 
validation  and  logical  or  output  validation 
definitions  laid  out  by  the  SIMVAL  n 
Proceedings.  Peer  review  results  pertaining  to  the 
structural  validity  of  the  model  appear  here  also. 

3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

These  are  DIS  W&A  Process  items  plus: 

•  Cost  of  additional  V&V  prior  to  the  point  of 
decision. 

•  Time  remaining  prior  to  the  point  of  decision. 

•  Risk  of  failure  and  impact  of  failure. 

•  Potential  costs  of  incorrect  decisions  (Type 
I/Type  n  errors). 

•  Results  of  peer  review  of  the  conceptual 
models. 

•  Other  credibility  related  information. 

It  cannot  be  overemphasized  that  accreditation  is 
a  management  decision  regarding  the  use  of  a 
particular  model  in  a  particular  application. 
Moreover,  it  should  be  for  use  with  a  particular 
data  set  that  has  itself  been  verified,  validated  and 
certified.  As  with  any  management  decision,  the 
information  is  rarely  complete,  rarely  absolute 
and  always  constrained  by  time  and  cost.  Thus, 
the  additional  factors  that  should  be  addressed 
throughout  the  accreditation  process  are: 


•  Cost  of  more  complete  and  better 
information. 

•  Time  remaining  to  collect  more  information. 

•  The  risks  and  costs  of  making  the  wrong 
decision. 

•  Accrediting  a  model  for  use  when  it  is,  in 
fact,  invalid  for  the  application. 

•  Refusing  accreditation  for  a  model  when  it  is, 
in  fact,  valid  for  the  application. 

•  All  credibility  related  data. 

With  regard  to  "credibility  related  data"  possibly 
the  single  most  powerful  tool  for  VV&A  is  peer 
review  of  the  conceptual  model.  Coding  and 
computerization  are  only  the  implementation  of 
the  conceptual  model.  No  matter  how 
sophisticated  the  coding  and  computerization 
techniques,  they  can  at  best  conserve  the  inherent 
value  of  the  conceptual  model.  Thus,  peer  review 
of  the  conceptual  model  can  offer  a  relatively  fast 
and  effective  "credibility  check"  in  the 
accreditation  process.  Peer  review,  by  definition, 
is  independent  of  the  developer  and  his/her 
organization. 

CONCLUSIONS 

The  WG  drew  several  conclusions. 

•  There  is  a  need  for  the  use  of  the  scientific 
method  and  sound  engineering  principles  for 
designing  distributed  interactive  simulation 
exercises  as  experiments  for  analysis. 

•  Peer  review  of  M/S  should  be  done 

•  It  is  important  to  use  MOE's  and  MOP's  in 
verification  and  validation  of  DIS  exercises 
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CHAPTER  7 
WORKING  GROUP  4B 

Advanced  Distributed  Simuiation/Distributed  Interactive 

Simuiation  (ADS/DIS) 

Co-chairs;  William  Dunn  and  Mr.  Rob  Wright 


1.  a.  Can  a  template  be  developed  to  support 
accreditation? 

We  first  addressed  the  question  "Can  a  template 
be  developed  to  support  an  accreditation?"  We 
agreed  that  the  answer  is  yes,  however  there 
should  be  a  family  of  templates  tailored  to  the 
"logical"  uses  of  DIS  applications.  A  typology  of 
logical  uses  of  DIS  is  shown  in  Figure  7-1.  It  was 
created  under  the  assumption  that  one  could 
define  general  classes  of  logical  use  such  that: 

•  The  role  of  DIS  in  any  reasonable  use  would 
appear  in  some  class. 

•  Validity  criteria  could  be  identified  for  each 
class. 

The  typology  first  distinguishes  between  using 
DIS  as  an  analytic  aid  and  stimulus. 

Assessments  of  the  validity  of  a  DIS 
configuration  must  be  based  upon  determination 
that  individual  participants  gained  the  desired 
experience.  The  means  for  doing  this  varies  with 
the  nature  of  the  experience  that  is  desired.  In 
training  for  standardized  proficiencies  where 
there  is  strong  transferability,  completely 
objective  means  for  testing  the  participants  exist. 
An  example  is  tank  gunnery  training  to  test 
individual  or  crew  proficiency.  When  training  for 
standardized  proficiencies  that  are  only  weakly 
transferable,  objective  criteria  exists,  but 
non-objective  means  are  required  to  assess 
whether  they  will  transfer  to  the  real  world 
experience.  Examples  include  applications  at  the 
battalion  level  using  Army  Training  and 
Evaluation  Program  Mission  Training  Plans  for 
evaluation  or  a  unit’s  mission  rehearsal.  For  those 


uses  where  there  are  no  standards,  assessments  of 
credibility  will  require  subjective  judgement  on 
the  part  of  appropriate  experts.  A  joint  staff 
training  exercise  is  an  example  of  a  use  requiring 
subjective  validation. 

Assessing  the  validity  of  a  DIS  configuration  as 
an  analytic  aid  depends  upon  the  degree  of 
predictivity  required  of  the  system  for  that  use. 
Non-predictive  uses  require  little  predictive 
power  from  DIS.  Hence,  for  these  uses  credibility 
depends  on  the  verification  that  the  system 
performs  the  logical  functions  expected  of  it  (be 
they  bookkeeping  or  logical  implications).  An 
example  of  bookkeeping  is  a  data  collection  effort 
of  hardware  reliability.  A  logic  tracing  example 
may  contrast  a  DIS  scenario  to  previous  lessons 
learned.  Strongly  predictive  uses  treat  system 
outputs  as  detailed  predictions  of  expected  real 
world  outcomes.  For  these  uses  credibility 
requires  statistically  rigorous  model  validation 
and  sufficient  test  cases  to  provide  significance  in 
the  results.  Examples  are  uses  of  doctrine 
validation  and  weather  effects  for  operations 
other  than  war.  Between  these  extremes  exists  a 
large  class  of  uses  that  require  weak  predictivity. 
For  these  uses,  credibility  depends  not  only  on 
system  characteristics,  but  on  the  details  of  the 
intended  analytic  argument  and  associated 
research  strategy.  Weak  predictivity  examples  are 
test  and  evaluation  analysis  using  simulation  to 
supplement  real  world  results  (a  fortiori)  while 
analysis  of  a  commander's  operations  plan  reflects 
an  example  of  plausible  outcomes. 

In  the  past,  the  credibility  of  simulation-based 
applications  has  been  assessed  through  a  process 
of  VV&A.  While  all  three  of  these  activities  are 
clearly  needed  in  the  context  of  DIS,  their  relative 
importance  varies  significantly  across  the 
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categories  of  logical  use.  For  non-predictive 
analysis,  credibility  can  be  established  primarily 
by  means  of  verification.  For  strongly  predictive 
analysis  and  uses  that  have  strongly  transferable 
standardized  proficiencies,  their  existence  means 
to  validate  the  system  or  training  protocol,  so  that 
this  validation  suffices  to  establish  credible  use. 
For  analyses  and  uses  that  are  either  non- 
standardized  or  weakly  transferable,  subjective 
judgement  is  unavoidable  in  assessing  credibility. 
Consequently,  it  is  for  these  uses  that  a 
formalized  process  of  accreditation  is  most 
needed. 

Early  in  the  planning  phase,  the  template 
development  process  should  ensure  application 
requirements  will  be  met.  The  first  step  is  for  the 
DIS  designer  to  review  and  analyze  the  purpose 
and  objectives  of  the  DIS  application.  The  role 
DIS  is  to  play  in  meeting  these  objectives  should 
be  defined  and  the  initial  allocation  of  the 
implementation  to  the  applicable  live,  virtual  or 
constructive  subset  should  be  decided.  This  is 
shown  as  Block  1  "Plan  Exercise  and  Develop 
Requirements"  in  the  VV&A  Process  Model, 
Figure  7-2.  After  considering  the  potential 
candidates  in  the  various  repositories  (e.g.,  M&S, 
data,  objects)  reflected  as  output  from  Block  2 
"Compliance  Standards  Verification  DIS 
Repository(ies)",  a  preliminary  design  can  be 
developed  into  a  conceptual  model. 
Development,  verification  and  validation  of  the 
conceptual  model  prior  to  construction  of  the 
actual  application  confederation,  reflected  in 
Blocks  3  and  4,  are  discussed  in  the  appendix  to 
this  subgroup  report. 

The  template  for  each  use  begins  by  tailoring  the 
library  of  elements  for  a  stand  alone  M&S. 
However,  the  DIS  template  must  go  further  to 
investigate  the  DIS-related  aspects  e.g.,  fidelity 
evaluation,  network  interfacing  and  performance, 
dead  reckoning  implementation,  resolution  of 
terrain  and  common  data  bases  and  aggregation 
issues  as  reflected  in  the  VV&A  Process  Model. 
The  fleshing  out  of  this  process  model  by  the 
M&S  community  is  ongoing  at  the  present  time 
and  will  drive  template  development. 


One  of  the  issues  to  be  fleshed  out  is  data 
verification,  validation  and  certification  (VV&C). 
VV&C  of  data  is  critical  to  the  entire  DIS  VV&A 
process  because  data  is  the  foundation  on  which 
the  entire  DIS  application  is  based.  Data  must  be 
collected  from  authoritative  sources  and  certified 
data  bases  whenever  possible.  An  authoritative 
source  is  an  organization  which  is  either: 

•  Designated  and  recognized  as  the  producer  of 
best-estimate  data  values. 

•  It  is  designated  to  conduct  producer  VV&C 
activities  for  one  or  more  categories  of  data. 

Certified  data  bases  are  sets  of  data  which  have 
been  verified,  validated  and  approved  (certified) 
for  specific  applications.  The  authoritative 
sources  should  construct  their  own  internal 
VV&C  of  the  data  before  providing  the  product  to 
the  DIS  designer. 

The  VV&A  agents  of  the  DIS  designer  (i.e.,  the 
user)  must  assess  the  data  sources  and 
modifications  to  the  data  to  meet  application 
unique  requirements.  In  some  cases  the  requisite 
data  may  not  be  available  from  a  recognized 
authoritative  source  or  may  not  be  acceptable  for 
the  DIS  application.  In  this  situation,  the  user  has 
three  options:  use  comparable  or  surrogate  data 
from  an  unapproved  source,  delete  the  algorithms 
or  simulations  requiring  the  data,  or  create/obtain 
the  data  and  conduct  a  complete  VV&C.  The 
user  must  determine  that  any  "unapproved  source" 
data  complies  with  applicable  data  standards.  If 
not,  then  a  decision  must  be  made  to  either 
modify  the  data  to  make  it  conform  or  accept  "as 
is"  and  accept  the  risk  associated.  Domain 
analysis,  data  activity  modeling,  data  element 
modeling,  and  data  standardization  are  all  very 
important  tools  available  to  assist  the  user  in  the 
W&CA^V&A  process.  Depending  on  the 
decisions  made  at  each  stage,  the  user  may  need 
to  use  some  or  all  of  these  tools  to  ensure  that  the 
data  is  suitably  evaluated  to  pass  the  W&C  tests 
necessary  to  reach  an  acceptable  level  of  risk  yet 
not  yield  an  adversarial  impact  on  cost,  schedule 
and  performance. 
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VV&A  template  development  must  include 
provisions  for  the  VV&C  of  data.  Each 
individual  M&S  will  have  previously  approved 
data  or  it  will  undergo  the  VV&CA^&A  steps 
necessary  to  use  the  data  before  being  accepted  as 
part  of  the  DIS.  V&V  agents  of  the  DIS  will 
conduct  data  for  VV&C  as  part  of  the  VV&A 
process  to  determine  that; 

•  The  data  has  been  properly  verified  and 
validated  for  use  with  each  of  the  individual 
M&S. 

•  The  each  individual  M&S  will  produce 
results  in  the  confederation  as  occurred 
independently  (i.e.,  that  the  participation  in 
the  DIS  will  not  affect  the  validity  of  the 
individual  M&S). 

2.  a.  How  do  we  present  the  V&V  status  of  a 
M&S? 

Our  subgroup  defined  this  to  mean  "How  do  we 
present  the  V&V  status  to  an  accreditor?"  To 
answer  this  question,  we  began  at  the  planning 
stage.-  When  an  application  is  planned,  the 
purpose  and  objectives  should  be  clearly  defined 
and  the  processes  determined  which  will  answer 
these  objectives.  The  V&V  status  for  the  DIS 
application  should  be  geared  toward  meeting  the 
objectives.  We  recommend  verification  status 
and  validation  status  be  reported  separately.  The 
status  is  qualitative  in  nature  and  addresses 
whether  the  objectives  have  been  met,  identifies 
any  limitations,  and  most  importantly  assesses 
risk  inherent  in  selecting  that  set  of  candidates. 
Recommendations  to  reduce,  eliminate  or  work 
around  the  risk  should  be  provided  to  the 
accreditor.  It  is  essential  that  these  V&V 
information  data  be  captured  and  placed  in  one  of 
the  DIS  repositories  for  future  reference. 

Appendix  G  in  Department  of  Army  Pamphlet  5- 
11  "Verification,  Validation,  and  Accreditation  of 
Army  M&S"  is  a  good  example  of  the  status 
reporting  elements  for  V&V.  These  are 
summarized  as; 


Executive  Summary.  This  stand-alone  section 
identifies  critical  issues,  trends  and/or 
sensitivities.  It  should  present  the  results  of  a 
reasonable,  systematic  examination  of  the  V&V 
process.  It  gives  an  objective  picture  of  the 
strengths,  weaknesses,  limitations  and  risk  in 
terms  of  the  intended  use.  A  specific  statement 
regarding  the  confidence  and  credibility 
associated  with  the  M&S  in  the  context  of  its 
intended  application  is  made  in  this  section. 

Overview  of  the  V&V  Planning; 

•  Identification  of  differences  from  V&V 
planned. 

•  Personnel  performing  the  VV&A. 

•  Description  of  the  verification  process  and/or 
tests. 

•  Logic  Verification  Results. 

•  Code  Verification  Results. 

•  Unresolved  Issues. 

•  Description  of  the  Validation  process  and/or 
tests. 

•  Evaluation  Criteria. 

•  Structural  Validation  Results. 

•  Output  Validation  Results. 

•  Unresolved  Issues. 

•  Ongoing  or  Follow-on  V&V  activities. 

•  References,  attachments  and  appendices. 

Although  we  were  emphatic  that  a  status  is 
qualitative,  an  interesting  sidelight  of  our 
subgroup  identified  the  "Strong  transferability" 
and  "Strongly  predictive"  legs  of  the  typology 
diagram.  Figure  7-1,  as  possibly  capable  of 
portraying  a  quantitative  status.  As  an  example,  a 
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tank  gunnery  application  could  show  percentage 
of  participants  who  tend  to  pass  proficiency  using 
this  DIS  application. 

3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its 
results)? 

The  next  question  is  "What  factors  should  we  use 
to  determine  whether  to  accept  a  M&S  (or  its 
results)?"  The  important  factors  are  V&V  status, 
risk  assessment,  schedule  and  programmatic 
considerations  which  may  affect  availability  for 
the  application,  cost,  ease  of  use  and  adaptability. 
V&V  status  is  important  since  this  gives  the 
accreditor  the  confidence  to  accept  or  reject  the 
DIS  application  for  its  intended  use.  The  V&V 
status  provides  the  accreditor  information  as  to 
how  thoroughly  and  rigorously  the  application 
was  reviewed.  This  knowledge  aids  in 
determining  that  the  application  has  been 
sufficiently  reviewed  by  qualified  subject  matter 
experts  thus  providing  confidence  to  the 
accreditor  and  reducing  risk. 

Risk  •  assessment  evaluates  and  measures  the 
amount  of  risk  quantitatively  in  terms  of  cost, 
schedule  and  performance.  Risk  factors  can  be 
selectively  considered  to  accurately  reflect 
application  unique  requirements  or  accreditation 
unique  criteria.  Similarly,  risk  avoidance  or  risk 
reduction  initiatives  can  also  be  addressed  in  the 
risk  assessment.  As  with  V&V  status,  a  well 
documented  and  accurately  structured  risk 
assessment  can  assist  the  accreditor  by  identifying 
high  risk  concerns  before  the  application 
progresses  to  the  final  decision  point. 

Schedule  implications  must  be  presented  at  every 
stage  of  the  decision  process  since  each  decision 
alternative  impacts  schedule  in  terms  of  cost, 
performance  or  availability. 

Programmatic  considerations  include 
management,  user  (e.g.,  training)  and  other  issues 
(e.g.,  human  factors,  safety).  Although  any  one  of 
these  issues  in  and  of  itself  may  not  be  a  "show 
stopper",  the  aggregation  of  these  issues  may 
increase  risk  of  failure  and  have  an  ultimately 


adverse  impact  on  cost,  schedule  or  performance. 
Programmatic  issues  must  be  addressed  in  the 
V&V  status. 

Less  important  was  the  source  of  the  model  and 
its  historical  use.  This  latter  element  should  be 
qualified  in  that  our  assumption  was  that 
individual  M&S  were  in  the  repository  (ies)  and 
have  a  Service/Agency  VV&A  history.  If  the 
M&S  was  used  in  a  stand  alone  mode,  its  history 
is  significantly  less  important  than  if  it  has  been 
used  in  previous  DIS  applications. 

4.  a.  What  is  a  legacy  model? 

The  last  question  concerned  "legacy"  models.  The 
subgroup  assumed  that  "legacy"  refers  to 
individual  M&S  in  the  repository  (ies)  since  we 
opined  that  for  most  applications,  it  will  be  rare 
for  an  entire  DIS  confederation  network  to  be 
reused  in  the  exact  same  configuration,  "legacy 
model"  is  defined  in  the  Defense  Systems 
Management  College  'Systems  Acquisition 
Manager's  Guide  for  the  Use  of  M&S'  as:  "A 
model  developed  in  the  past  which  is  still  in  use 
that  was  not  implemented  using  today's  standards 
(e.g.,  software,  communication,  DIS,  ALSP,  etc.) 
Some  legacy  models  have  been  modified  with 
interfaces  to  some  of  the  current  standards 
extending  their  usefulness  and  interoperability 
with  newer,  standards  based  models." 

We  determined  that  individual  legacy  M&S  was 
not  an  issue  for  DIS  applications  since  the  basic 
standards  to  be  in  the  repository  required  each 
M&S  to  have  a  Service/Agency  VV&A  history 
and  meet  DIS-compliance  testing.  Thus,  they  are 
to  receive  VV&A  in  the  same  manner  as  other 
M&S.  Finally,  any  M&S  (whether  considered  as 
an  individual  legacy  M&S  or  otherwise)  will 
acquire  credibility  by  use  in  a  DIS  environment 
assuming  it  meets  DIS  standards  just  as  its  usage 
would  in  a  stand  alone  case. 
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CHAPTER  8 

WORKSHOP  SUMMARY 


The  findings  of  the  SIMVAL  1994  working 
groups  are  summarized  in  this  chapter.  The 
summary  is  organized  by  each  key  question. 

1.  ACCREDITATION  TEMPLATE 

1.  a.  Can  a  template  be  developed  to 
support  an  accreditation? 

All  working  groups  said  Yes.  Most  indicated 
that  there  should  be  a  family  of  templates  based 
on  M&S  level/application  type/use.  The 
concern  was  that  one  template  would  not  fit  all 
M&S  uses.  A  super-template'  might  be  created, 
but  it  would  have  to  be  tailored  for  each 
accreditation  by  application  type. 

1.  b.  What  would  such  a  template  look 
like? 

The  following  is  a  composite  from  all  working 
groups  of  the  items  to  be  included  in  the 
template. 

A.  Application  Description 

1.  Purpose  of  application 

2.  Objectives 

3.  Specific  issues/concems 

4.  Application  risk/importance 

5.  Time  requirement  for  accreditation 

6.  Resource  constraints 

B.  Accreditation  Basics 

1.  M&S  Description  (Title,  M&S 
scope/overview,  version) 

2.  M&S  Proponent 
(Reputation/Experience) 

3.  Proposed  use  in  Decision  Process 
(Integration  with  other  methods/data) 

4.  Key  Functional  Representations 

5.  V&V  Status  (See  response  to  question 

2.a) 

•  Includes  Credentials  of  V&V  agents 


6.  Sensitivity  Analyses  Performed 

7.  Known  Limitations/Constraints 

•  Includes  Degree  of  Clarity  (Cause 
and  Effect) 

8.  Configuration  Management/Control 

9.  Documentation  Available 

10.  Data 

a.  Sources 

b.  Verification  Status 

c.  Validation  Status 

d.  Certification 

e.  Uncertainty 

11.  Affordability 

12.  Availability  (of  M&S  and  Trained, 
Experienced  Users) 

13.  Risk  Reduction  Actions 

14.  Alternatives  considered 

15.  Historical  Use,  Acceptance,  Known 
Applications  (including  any  specific 
DIS  use) 

16.  Previous  Accreditation  History 
(including  any  specific  DIS  use) 

C.  Assessment  Approach 

1.  Acceptance  criteria  (by  key  function) 

2.  Priority  Sequence  of  key  functions  for 
V&V  (useful  if  under  time  or  resource 
constraints) 

3.  Methodology  for  Meeting  Criteria 

a.  Verification  Plan 

b.  Validation  Plan 

c.  DIS  Considerations  (if  a  Distributed 
Interactive  Simulation) 

•  See  DIS  Workshop  W&A 
Process 

d.  Impact  on  application  of  M&S 
functions  not  verified. 

e.  Impact  on  application  of  M&S 
functions  not  validated. 

D.  Recommendations/Options 

1 .  Actions  to  be  Taken  (if  any) 

2.  Caveats,  by  Functional  Representation 
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E.  Decision 

Other  models  or  information  sources  for  the 
templates  suggested  were  the  SMART 
(Suspectibility  Model  Assessment  and  Range 
Test)  set,  DIS  Workshop  VV&A  Process,  DA 
PAM  5-11,  SIMTAX,  DMSO  IPL#2,  and  Paul 
Davis/Rand  Model. 

1.  c.  How  should  we  go  about  developing 

such  a  template? 

The  following  approach  came  from  the  working 
groups  and  was  the  consensus  approach  of  the 
Co-chairs/SAG  meeting  held  immediately  after 
the  Workshop  ended. 

The  output  of  the  SIMVAL  1994  Workshop  will 
be  used  as  the  basis  for  developing  a  draft  set  of 
templates.  The  result  would  be  a  set  of 
guidelines  for  specific  applications  and  M&S 
types.  These  draft  guidelines  will  be  put  on  an 
electronic  bulletin  board  now  being 
implemented  by  MORS.  This  would  allow 
review  and  comments  from  across  the 
community  on  the  form/contents  of  these 
templates.  When  the  draft  has  been  reviewed 
(and  comments  die  down),  the  final  set  will  be 
published. 

2.  V&VSTATUS 

2.  a.  How  do  we  present  the  V&V  status  of 
a  M&S? 

The  working  groups  generally  agreed  that  V&V 
status  should  be  reported  separately. 

A.  Strategic  Indicators 

1.  Age/maturity 

2.  Level  of  Configuration  Management 

3.  Size  of  User  Group 

4.  Major  versions/Pace  of  change 

5.  Number  of  applications/studies 
supported 

6.  Data  Sources 

7.  Functional  Balance 

8.  Functional  Adaptability 


B.  Verification  Status 

1 .  Key  Functions/Elements/Interactions  List 

2.  By  key  function/element/interaction 

a.  Verification  method  used 

b.  M&S  version  to  which  applied 
•  Include  linkage  to  current 

version 

c.  Date  of  application 

d.  Verification  agent 

e.  Verification  report  reference 

f.  Verification  results  (quantitative 
statements) 

g.  Verification  conclusions 

(qualitative  statements) 

C.  Validation  Status 

1 .  Key  Functions/Elements/Interactions  List 

2.  By  key  function/element/interaction, 

a.  Validation  method  used 

b.  M&S  version  to  which  applied 

•  Include  linkage  to  current  version 

c.  Date  of  application 

d.  Validation  agent 

e.  Validation  report  reference 

f.  Validation  results  (quantitative 
statements) 

g.  Validation  conclusions  (qualitative 
statements) 

D.  Limitations/Risks 

2.  b.  Are  there  "units"  for  V&V? 

All  the  working  groups  agreed  there  were  no 
"units’ for  V&V. 

3.  ACCREDITATION  FACTORS 

3.  What  factors  should  we  use  to  determine 
whether  to  accept  a  M&S  (or  its  results)? 

The  working  groups  indicated  the  factors  used 
in  the  accreditation  template  were  appropriate 
(See  response  to  question  IB.  above). 
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4.  LEGACY  M&S 

4.  a.  What  is  a  legacy  model? 

The  working  groups  varied  in  their  response  to 
this  question. 

•  "[A  legacy  M&S]  has  a  long  history  of  use; 
high/moderate  level  of  acceptability; 
negligible  V&V." 

•  "A  legacy  model  is  continually  improved 
over  time,  configuration  managed,  and  has 
broad'  acceptance." 

•  "A  model  developed  in  the  past  that  is  still 
available  and  that  was  not  implemented 
using  current  standards." 

•  "Assume  legacy'  refers  to  individual  M&S 
in  the  DIS  repository.  Non-issue  for  DIS 
environment  since  M&S  must  meet  basic 
DIS  standards  to  be  in  the  repository." 

•  '"Legacy'  is  not  a  useful  term." 

4.  b.  Does  a  "legacy”  M&S  acquire  any  V&V 
credibility  by  its  use  and  acceptance? 

The  working  groups  basically  agreed  credibility 
accrued  to  a  M&S  by  use  (for  a  legacy'  or  any 
other  M&S),  but  should  go  through  the  same 
processes  for  W&A  as  any  other  M&S.  It 
should  have  a  larger  database  of  historical  use 
and  W&A  from  which  to  draw,  thereby  making 
an  accreditation  decision  easier  or  quicker. 


GENERAL 

Most  of  the  working  groups  mentioned  that  the 
existence  of  repositories  of  M&S  information 
and  make  VV&A  less  expensive.  It  would  also 
eliminate  redundant  V&V  efforts.  The 
recommendation  was  not  necessarily  for  a  single 
repository,  but  at  least  for  common  standards  for 
all  DoD  repositories  so  that  information  is 
compatible  and  consistent. 
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CHAPTER  9 

WHERE  DO  WE  GO  FROM  HERE? 


There  was  a  SMVAL  SAG  Meeting  held 
immediately  following  the  SIMVAL  ’94 
Workshop  to  determine  where/how  the  SIMVAL 
series  should  proceed.  The  guidance  provided 
was: 

A.  Use  the  WG  findings  to  develop  a  set  of  draft 
accreditation  templates  and  guidelines  for  how 
much  V&V  is’  needed  to  permit  a  credible 
accreditation  decision.  The  guidelines  will 
consider  application  type,  application 
importance/risk  and  M&S  type.  The  templates 
and  guidelines  will  be  put  on  the  MORS 
electronic  bulletin  board  to  provide  MORS 
members  the  opportunity  to  review,  comment  and 
suggest  changes.  At  the  point  where  there 
appears  to  be  general  community  consensus,  they 
will  be  formally  published. 

B.  It  was  agreed  that  no  additional  SIMVAL 
workshops  would  be  held  until  the  community  has 
had  time  to  use  the  VV&A  guidelines  developed 
by  the  SIMVAL  series.  At  some  future  date, 
another  workshop  to  discuss  case  studies  of  the 
guidelines  would  be  beneficial. 

C.  The  SIMVAL  series  should  sponsor  an 
activity  to  consider  simulation  validation  in 
commercial  industry.  There  are  a  number  of 
commercial  systems  (e.g.,  aircraft,  tmcks)  similar 
to  military  systems,  that  are  modeled  by  industry 
for  development  and  other  purposes. 
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GLOSSARY  OF  ABBREVIATIONS 
AND  ACRONYMS 


ADS 

Advanced  Distributed  Simulation 

AH 

Air  Force  Instruction 

ALSP 

Aggregate  Level  Simulation  Protocol 

AMSAA 

Army  Materiel  and  Systems  Analysis  Activity 

ASD 

Assistant  Secretary  of  Defense 

BMDO 

Ballistic  Missile  Defense  Organization 

C/M 

Configuration  Management 

DIS 

Distributed  Interactive  Simulation 

DMSO 

Defense  Modeling  and  Simulation  Office 

DoD 

Department  of  Defense 

DUSA(OR) 

Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

ECM 

Electronic  Countermeasures 

ECCM 

Electronic  Counter-Countermeasures 

GAO 

Government  Accounting  Office 

IDA 

Institute  for  Defense  Analyses 

IPL 

Integrated  Priority  List 

IR 

Infrared 

IV&V 

Independent  Verification  and  Validation 

JHU/APL 

Johns  Hopkins  University/Applied  Physics  Laboratory 

M&S 

Models  and  Simulations 

MOE' 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

MORS 

Military  Operations  Research  Society 

NADC 

Naval  Air  Development  Center 

OSD 

Office  of  the  Secretary  of  Defense 

PAM 

Pamphlet 

PDU 

Protocol  Data  Unit  (DIS) 

SAG 

Senior  Advisory  Group 

SIMTAX 

Simulation  Taxonomy 

SIMVAL 

Simulation  Validation 

SURVIAC 

Surviveability  Integrated  Activity  Center 

v&v 

Verification  and  Validation 

VV&A 

Verification,  Validation  and  Accreditation 
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SIMVAL  94'  WORKSHOP  AGENDA 

Draft— August  3, 1994 


Tuesday,  September  27 

1 600  Early  registration  Begins  (Lobby  of  Fred  Harvey  Hotel) 
1800  End  Early  registration 


Wednesday,  September  28 

0700  Registration/Continentai  Breakfast  (BDM  Sandia  Vista  Conference  Center) 
Cochair's  Breakfast  Meeting  and  Orientation  (BDM  SVCC) 

0800  Welcome  by  MORS  Representative 

0810  Introduction  to  Workshop  (Williams/Sikora) 

Background  of  SIMVAL  Series 

Overview  of  SIMVAL  94'  Workshop 

-  Purpose 

-  Agenda/Key  Players 

-  Breakout  rooms 

-  Products 

Administrative  Announcements 
0830  Plenary  Session  1 

Air  Force  Overview  of  VV8A — Process,  Issues  and  Concerns 
Air  Force  Exemplary  Case  Study/Paper 

1000  Break 

1015  Army  Overview  of  W&A — Process,  Issues  and  Concerns 
Army  Exemplary  Case  Study/Paper 

1145  Lunch 

1300  Plenary  Session  2 

Navy  Overview  of  W&A — Process,  issues  and  Concerns 
Navy  Exemplary  Case  Study/Paper 

1430  Break 

1445  Marine  Corps  Overview  of  W&A — Process,  Issues  and  Concerns 
1 51 5  OSD  Overview  of  W&A— Process,  Issues  and  Concerns  (Sanders) 

1 545  GAO  Overview  of  W&A — Process,  Issues  and  Concerns 
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Wednesday,  September  28  (cont.) 

1615  End  Plenary  Session  II 

Overview  of  Working  Group  Expectations 

1 630  Working  Group  Introductions/Orientations 

Break-up  into  Working  Groups — (Assigned  Working  Group  Areas) 

1 700  End  Working  Group  Introductions/Orientations 

Begin  Mixer  (Lobby  ofSandia  Vista  Conference  Center) 

1830  End  Mixer 

Thursday,  September  29 

0730  Continental  Breakfast  (Lobby  ofSandia  Vista  and  Randolph  Conference  Ctr) 

0800  Working  Group  Sessions  (Breakout  Rooms) 

1200  Lunch  (Lobby  ofSandia  Vista  Conference  Center) 

1 300  Working  Group  Sessions  (Breakout  Rooms) 

1700  End 

(Breakout  Rooms  available  for  evening  sessions  to  prepare  session  summan'es  until  2100) 

Friday,  September  30 

0730  Continental  Breakfast  (Lobby  ofSandia  Vista  Conference  Center) 

0815  Working  Group  1  Summary  (Sandia  Vista  Conference  Center) 

0845  Working  Group  2  Summary  (Sandia  Vista  Conference  Center) 

0915  Break 

0945  Working  Group  3  Summary  (Sandia  Vista  Conference  Center) 

1015  Working  Group  4  Summary  (Sandia  Vista  Conference  Center) 

1045  Break 

1100  Workshop  Summary  (Sandia  Vista  Conference  Center) 

1130  End  Workshop 

1200  SAG/Working  Group  Cochairs  Meeting — How  Do  We  Proceed  From  Here? 
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TERMS  OF  REFERENCE 

MORS  SIMULATION  VALIDATION  SERIES  (SIMVAL) 
1994  WORKSHOP  ON  W&A 


Background 

The  MORS  Simulation  Validation  Series  (SIMVAL)  consists  of  workshops,  mini¬ 
symposia,  advisory  group  activities,  and  reports  (proceedings)  with  the  punxse  of 
reviewing  areas  of  verification,  validation  and  accreditation  (VV&A),  determining  a 
consistent  set  of  definitions,  developing  a  general  structure  and  examining  the  current 
methodologies.  The  activities  (ongoing  since  1990)  are  directed  by  a  Senior  Advisory 
Group  (SAG)  which  is  composed  of  senior  analysts  and  executives  representing  a 
breadth  of  simulation  experience. 

The  results  of  the  SIMVAL  series  has  been  a  consistent  set  of  definitions  for 
W&A,  a  general  structure  for  VV&A  in  modeiy^imulation  development,  and  grouping 
and  descriptions  of  verification  and  validation  methodologies.  These  results  are 
documented  in  “Simulation  Validation  Workshop  Proceedings  (SIMVAL  II)”.  A  copy  of 
these  proceeding  will  be  provided  to  SIMVAL  94  attendees  as  a  part  of  a  read-ahead 
package. 


Objectives 

The  SIMVAL  94  workshop  will  address  the  theme  question  “How  Much  V&V  is 
Enough?”.  The  objectives  of  the  workshop  are  to  (1)  discuss  Service  issues  in 
defining  and  implementing  W&A  policy,  including  “how  much  W&A  is  enough”;  (2) 
within  the  context  of  the  different  levels  of  models  and  simulation  (M&S),  review  a 
limited  number  of  case  studies  in  applying  W&A  to  discuss  the  approach  and  lessons 
learned;  (3)  use  these  lessons  learned  to  develop  an  initial  draft  set  of  guidelines  or 
templates  to  guide  field  W&A  efforts.  Findings  will  be  documented  in  an  addendum 
to  the  MORS  W&A  report  or  as  a  separate  report  as  appropriate. 

The  focus  of  the  workshop  will  be  on  an  exchange  of  information  and  ideas  on 
how  to  develop  and  implement  practical  VV&A  procedures.  Guidelines  will  be 
developed  based  on  successes  and  failures  in  attempting  to  define  and  implement 
policy  across  the  Services  and  across  different  levels  of  models. 

Agenda 

The  agenda  will  include  an  introductory  session  describing  the  objectives  and 
scope  of  the  workshop  and  the  results  of  SIMVAL  to  date.  This  will  be  followed  by  a 
plenary  session  with  briefings  on  each  Services’  W&A  policy.  A  panel  composed  of 
Service  and  OSD  senior  executives  will  then  discuss  their  experiences  in  developing 
and  applying  W&A  policy. 
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Selected  papers  will  be  presented  on  approaches  and  methodology  for 
determining  “how  much  V&V  is  enough”.  Topics  will  include  organizational 
approaches  for  accreditation,  levels  of  accreditation,  Service  W&A  guidelines,  etc. 

On  the  second  day,  the  workshop  will  break  into  five  individual  sessions 
pertaining  to  Engineering  level  models,  engagement  level  models,  mission  level 
models,  campaign  level  models,  and  Distributed  Interactive  Simulation. 

Each  group  will  focus  on  broad  W&A  issues  as  well  as  issues  specific  to  the 
level  of  M&S  being  considered  by  that  group.  Group  co-chairs  will  invite 
presentations  op  W&A  case  studies  and  organizational  approaches  to  W&A. 

Topics  will  include; 

1 .  Feasibility  and  approach  to  developing  W&A  “templates”.  The  purpose  of 

the  templates  would  be  to  guide  users  in  determining  the  level  of  V&V  required  for 
different  circumstances.  Potential  template  categorization  could  be  a  function  of  the 
type  of  decision  (test  planning,  sensitivity  analysis,  acquisition  decision, ... ),  type  of 
model  (engineering,  mission, ... ),  source  of  model  (new  code,  revision  of  old  code, 
use  of  existing  model . ),  etc. 

2.  V&V  of  “legacy”  models.  How  much  V&V  is  enough  for  models  that  have 
been  in  use  and  accepted  by  the  community? 

3.  How  can  the  V&V  status  of  a  model  be  presented? 

4.  What  factors  can  be  used  to  determine  whether  a  model  is  acceptable  for  a 
specific  application? 

The  morning  of  the  third  day  will  be  devoted  to  summarizing  major  findings  of 
each  working  group.  Findings  will  be  referred  by  the  SAG  to  determine  future  actions, 
such  as  additional  SIMVAL  efforts  to  test  and  further  refine  W&A  templates. 

The  workshop  will  end  at  noon  30  September.  The  Senior  Advisory  Group  will 
meet  until  1700  30  September. 


Membership 

The  workshop  co-chairs  will  be  Dr.  Marion  Williams,  FS,  of  the  Air  Force 
Operational  Test  and  Evaluation  Center  and  Mr.  James  Sikora  of  BDM  International 
The  chairs  will  control  the  membership  so  that  it  falls  in  the  range  of  75-125.  Active 
use  will  be  made  of  member  of  the  appropriate  MORS  working  groups,  and 
participants  in  previous  SIMVAL  activities  will  be  encouraged  to  continue  their 
participation.  From  the  applicants  responding  to  the  announcement,  priority  will  be 
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given  to  those  with  the  most  experience  in  W&A. 

Session  co-chairs  will  moderate  session  discussions  and  will  take  responsibility 
for  preparing  and  presenting  their  session  summaries.  The  session  co-chair  will  also  be 
responsible  for  writing  the  segment  of  the  MORS  SIMVAL  report. 

The  following  individuals  have  agreed  to  co-chair  the  individual  sessions: 


Campaign  Level  Models 

Lt  Col  Robert  Sheldon,  USAF 

Mr.  Howard  Whitley 

Mission  Level  Models 

Ms.  Cheryl  Black 

Mr.  John  Riente 

Engagement  Level  Models 

3A  Mr.  Robert  Hartling 

Mr.  Mark  Ralston 

3B  Dr.  Paul  Muessig 

Mr  Michael  Borowski 

Engineering  Level  Models 

Dr.  Julian  Palmore 

Mr.  Robert  Eberth 

Distributed  Interactive  Simulation 

Mr.  William  Dunn 

Mr.  Rob  Wright 

Schedule  and  Fees 

The  workshop  will  be  held  in  Albuquerque,  NM  at  BDM  International,  Sandia 
Vista  Conference  Center  September  28-30, 1994, 

There  will  be  a  registration  fee  of  $150  for  Federal  Government  employees  and 
$300  for  all  others. 
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Military  Operations  Research  Society  Workshop: 

SIMVAL  ^94 

Attendee  List  (09/29/94) 


Joseph  A.  Adamo 

McDonnell  Douglas  Aerospace  Co 

PO  Box  516,  MC  3065146 

St.  Louis  MO  63166 

OFF  TEL:  (314) -233-2688 

FAX:  (314)-234-4128 


MAJ  Donald  J.  Adamson 

31  TES  OL-KT 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117 

OFF  TEL:  ( 505 ) -846-2331  DSN:  246-2331 

FAX:  (505)-846-4285 

E-mail:  adamsond@p2.afotec.af.mil 


Gerald  P.  Alldredge 
BDM  International 
1801  RANDOLPH  S.E. 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -848-5432 
FAX:  (505) -848-4167 


DR  David  E  Anderson 

HQ  USAF/AFSAA 

1570  Air  Force,  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  (703 ) -695-0725  DSN:  225-0725 

FAX:  (703)-697-3441 

E-mail :  ander son@af saa . HQ . AF . MIL 


DR  John  V  Adams 
IDA 

1801  N.  Beauregard  Street 
Alexandria  VA  22311-1772 
OFF  TEL:  ( 703 ) -845-2475 
FAX:  (703)-845-6722 


Natalie  S  Addison 

Military  Operations  Research  Society 
101  S  Whiting  Street 
Suite  202 

Alexandria  VA  22304-3483 
OFF  TEL:  ( 703 ) -751-7290 
FAX:  (703)-751-8171 


COL  Thomas  L.  Allen 

Air  Force  Studies  and  Analyses  Agency 
AFSAA/CC 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  (703) -695-9046  DSN:  225-9046 

FAX:  (703) -697-3441 

E-mail:  allentc@afsaa.hq.af.mil 


DR  Richard  S  Baty 
BDM  International 
1801  Randolph  Rd,  SE 
Albuquercjue  NM  87106 
OFF  TEL:  ( 505 ) -848-5337 
FAX:  (505) -848-5248 
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Francoise  Becker 

Center  for  Naval  Analyses 

4401  Ford  Avenue 

Alexandria  VA  22302-0268 

OFF  TEL:  ( 703 ) -824-2526  DSN:  289-2638 

FAX:  (703)-824-2949 

E-mail:  beckerf@cna.org 


DR  Bruce  W  Bennett 
RAND 

1700  Main  Street 
Santa  Monica  CA  90407-2138 
OFF  TEL:  ( 310 ) -393-0411 
Ext:  00000667 
FAX:  (310)-451-7038 
E-mail :  bruce . bennett@rand.org 


Vernon  M  Bettencourt  Jr 
The  MITRE  Corporation 
Synthetic  Environments  Apps  Dept 
7525  Colshire  Drive,  MS  W641 
McLean  VA  22102-3481 
OFF  TEL:  (703 ) -883-7364 
FAX:  (703)-883-1370 
E-mail:  vbetten@mitre.org 


DR  Gregory  J  Born 
RAND 

1700  Main  Street 
FOB  2138 

Santa  Monica  CA  90407-2138 

OFF  TEL:  (310) -393-0411 

Ext:  00000675 

FAX:  (310)-393-4818 

E-mail :  gregory_born@rand . org 

Edward  C  Brady  FS 
Strategic  Perspectives,  Inc. 
7704  Lakeloft  Court 
Fairfax  Station  VA  22039 
OFF  TEL:  (703) -250-6338 
FAX:  (703)-250-3637 


DR  Gerry  H  Cabaniss 
BDM  International 
MS  R14 

1801  Randolph  Road 

Albuquerque  NM  87106 

OFF  TEL:  ( 505 ) -848-5230 

FAX:  (505)-848-4167 

E-mail:  gcabaniss@lan.mcl.bdm.com 


Cheryl  Black 
HQ  AFOTEC/SAN 
8500  Gibson  Blvd. ,  SE 
Kirtland  AFB  NM  87117 
OFF  TEL:  ( 505 ) -846-0503 
FAX:  (505) -846-5145 


LCDR  Michael  Borowski 

COMOPTEVFOR 

7970  Diven  Street 

Norfolk  VA  23505-1498 

OFF  TEL:  ( 804 ) -445-0292  DSN:  565-0292 

FAX:  (804)-445-8516 

E-mail :  borowski@tecnet 1 .jcte.jcs.mil 


David  J  Brown 
Mitre  Corporation 
7525  Colshire  Drive 
McLean  VA  22102 
OFF  TEL:  (703 ) -883-5833 
FAX:  (703) -883-1370 


COL  John  D  Carlile 
JTMD/AO 

1951  2nd  Street,  SE 

Kirtland  AFB  NM  87117-5559 

OFF  TEL:  ( 505 ) -846-1472  DSN:  246-1472 

FAX:  (505)-846-1486 
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Jesus  E  Carrillo 

Research  Analysis  and  Maintenance,  Inc, 

1790  Lee  Trevino  Dr,  #600 

El  Paso  TX  79936-4525 

OFF  TEL:  (915 ) -592-7047 

Ext:  00000013 

FAX:  (915)-595-0559 


DR  Edward  S  Cavin 

Center  for  Naval  Analyses 

PO  Box  16268 

4401  Ford  Avenue 

Alexandria  VA  22302-0268 

OFF  TEL:  (703 ) -824-2951  DSN:  289-2638 

FAX:  (703)-824-2949 

E-mail :  ccavin@cna . org 


LTC  George  T  Cherolis 

FKJ3-PL-OA 

PSC  303,  Box  27 

Unit  15237 

APO  AP  96204-0027 

OFF  TEL:  (011 ) -822-7913  DSN:  723-8371 

Ext:  00000824 

FAX:  (011)-822-8244 

E-mail:  cfcd-pl-oa@emh7 . korea.army.mil 
Julie  C  Chu 

US  Army  Armament  Rsch  Dev  &  Engineer  Ctr 
ATTN;  SMCAR-ASH 

Picatinny  Arsenal  NJ  07806-5000 
OFF  TEL:  (201)-724-7088  DSN:  880-7088 
FAX:  {201)-724-4111 
E-mail:  jchu@pica.army.mil 
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Helga  Carter 
HQ  AFOTEC 

Kirtland  AFB  NM  87117 


LtCol  Kevin  E  Cheek 
AFSAA/SAG 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  ( 703 ) -614-4247  DSN:  224-4247 

FAX:  (703)-697-1226 

E-mail :  cheek@afsaa.hg.af.mil 


Willard  M  Christenson 

Institute  for  Defense  Analyses 

SFRD  Room  517 

1801  N.  Beauregard  St 

Alexandria  VA  22311 

OFF  TEL:  (703 ) -845-2439 

FAX:  (703)-845-2255 

E-mail :  wchr istegida . org 


Gary  Q  Coe 
ANSER 

1215  Jefferson  Davis  Hwy 
Suite  800 

Arlington  VA  22202 
OFF  TEL:  ( 703 ) -416-3073 
FAX:  (703)-416-3389 
E-mail :  coeg@ANSER . org 


Allan  L  Collie 
BDM  International 
1801  Randolph  Road,  SE 
Albuquerque  NM  87106 
OFF  TEL:  (505 ) -848-5231 
FAX:  (505)-848-5809 


Capt  Angie  Crawford 
AFOTEC/ SAN 

Kirtland  AFB  NM  87117 

OFF  TEL:  ( 505 ) -846-1377  DSN:  246-1377 

FAX :  ( 505 ) -846-5145 

E-mail :  crawf oraQpl . af otec . af.mil 
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Kevin  R  Crosthwaite 
Booz  Allen  &  Hamilton 
4141  Colonel  Glenn  Hwy 
Suite  131 
Dayton  OH  45431 
OFF  TEL:  ( 513 ) -429-9509 
FAX:  (513)-429-9795 


Joseph  J  Cynamon 
The  MITRE  Corporation 
C3I  Division,  D-46  MS  1302 
202  Burlington  Post  Road 
Bedford  MA  01730 

OFF  TEL:  ( 617 ) -377-5550  DSN:  478-5550 
FAX:  {617)-271-7469 

E-mail :  joe_cynamon . masc@qrnbase . mitre . org 


Keith  P  Curtis 
The  MITRE  Corporation 
7525  Colshire  Dr 
MS  Z464 

McLean  VA  22102-3481 
OFF  TEL:  ( 703 ) -883-7905 
FAX:  (703)-883-1379 
E-mail :  kcurt is@mitre . org 


Michael  H  Davis 
Det  4  505  CCEG 
Room  210 

1655  First  Street,  SE 

Kirtland  AFB  NM  87117-5617 

OFF  TEL:  ( 505 ) -846-2391  DSN:  246-2391 

FAX:  (505)-846-1486 


Vincent  E  Debban 

Booz,  Allen  and  Hamilton 

Suite  600 

1953  Gallows  Road 

Vienna  VA  22182 

OFF  TEL:  (703)-902-6953 

FAX:  (703) -902-7171 


Carrol  R.  Denney 

TRADOC  Analysis  Center-WSMR 

Attn:  ATRC-WEB 

White  Sands  Missile  Range  NM  88002-5502 
OFF  TEL:  ( 505 ) -678-6778  DSN:  258-6778 
FAX:  (505)-678-5104 

E-mail :  denneyc@wsmr-emh91 . army . mil 


DR  James  A  Dewar 
RAND 

1700  Main  Street 

Santa  Monica  CA  90407-2138 

OFF  TEL:  (213 ) -393-0411 

Ext:  00000755 

FAX:  (213)-393-4818 

E-mail :  dewar@rand . org 


George  T  Dietrich  Jr 
Booz  Allen  &  Hamilton 
Suite  300 

2201  Buena  Vista  Drive,  SE 
Albuquerque  NM  87111 
OFF  TEL:  (505 )-247-8722 
FAX:  (505)-242-4821 
E-mail:  DIETRICH  GEORGE@bah.com 


Robert  D  Dighton 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  Street 

Alexandria  VA  22311-1772 

OFF  TEL:  ( 703 ) -845-6992 

FAX:  (703)-845-6911 

E-mail :  rdighton® ida . org 


DR  Henry  C  Dubin 

US  Army  Operational  Test  &  Eval  Comd 
Attn;  CSTE-2T 
4501  Ford  Avenue 
Alexandria  VA  22302-1458 
OFF  TEL:  ( 703 ) -756-2367  DSN:  289-2367 
FAX:  (703)-756-0779 

E-mail :  dubin%textl@texcom-emhl . army.mil 
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Capt  Ross  A.  Dudley 

31  TES,  OL-KT 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117--5558 

OFF  TEL:  ( 505 ) -846-5328  DSN:  246-5328 

FAX:  (505)-846-4285 

E-mail :  dudleyr@p2 . af otec • af . mil 


James  B  Duff 
COMOPTEVFOR 

Technical  Director,  Code  OOT 
7970  Diven  Street 
Norfolk  VA  23505-1498 

OFF  TEL:  (804) -444-5197  DSN:  564-5197 
FAX:  (804)-445-9174 
E-mail :  duf f@tecnetl •jcte.jcs.mil 


William  H  Dunn 
US  Army  MISMA 
Crystal  Square  2,  #808 
1725  Jefferson  Davis  Hwy 
Arlington  VA  22202 

OFF  TEL:  ( 703 ) -607-3384  DSN;  327-3384 
FAX:  (703)-607-3381 

E-mail :  william. h . dunn@pentagon-ldmsl8 . 


Robert  A  Eberhard 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  Street 

Alexandria  VA  22311 

OFF  TEL:  (703 ) -845-6939 

FAX:  (703)-845-6911 

E-mail:  <reberhar@oed-ul . ida.org 

.mil 


Robert  W  Eberth 

Palmer-Eberth 

Suite  115-212 

1350  Beverly  Road 

McLean  VA  22101 

OFF  TEL:  (703 ) -769-3325 

FAX:  (703) -685-9849 


DR  Henry  L  Eskew 
Center  for  Naval  Analyses 
4401  Ford  Ave 
Alexandria  VA  22302 

OFF  TEL:  (703) -824-2254  DSN:  289-2638 
FAX:  (703)-824-2949 


LtCol  Coral  Fallstead 

AFOTEC/OL-PC 

National  Test  Facility 

730  Irwin  Ave,  Room  2309-1 

Falcon  AFB  CO  80912-7300 

OFF  TEL:  (719 )-380-3695  DSN;  749-3695 


DR  Sharon  Ellis 
ENTER,  Inc 
2201  Buena  Vista,  SE 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -242-5860 
FAX:  (505) -242-7034 


Joseph  L  Faix 
TRW 

JTMD/AO 

1951  2nd  Street  SE 

Kirtland  AFB  NM  87117-5559 

OFF  TEL:  ( 505 ) -846-6846  DSN:  246-6846 

FAX:  (505) -846-6843 


Peter  D.  Feuerstein 

US  Army  Space  &  Strategic  Defense  Cmd 
Attn:  CSSD-SA-T 

105  Colt  Circle 
Huntsville  AL  35811 

OFF  TEL:  (205 ) -955-1682  DSN:  645-1682 
FAX:  (209)-955-4339 
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Christine  A  Fossett 
US  GAO 

Office  of  Policy,  Room  6800 
441  G  Street,  NW 
Washington  DC  20548 
OFF  TEL:  (202 ) -512-2956 
FAX:  (202)-512-4844 


Peggy  Gordon 

HQ  USAF/XOMT 

1480  Air  Force  Pentagon 

Washington  DC  20330-1480 

OFF  TEL:  (202 >-504-5339  DSN:  285-5339 

FAX:  (202)-504-5352 

E-mail:  pgordon@xom-mail.hg.af.mil 


Jimmy  R  Grussmeyer 
SRS  Technologies 
Washington  Operations  Div 
1500  Wilson  Blvd,  #800 
Arlington  VA  22209 
OFF  TEL:  (703 ) -522-5588 
FAX:  (703)-527-3807 


David  H  Hall 

Naval  Air  Warfare  Center 

Weapons  Div,  Code  C21805 

1  Administration  Circle 

China  Lake  CA  93555-6001 

OFF  TEL:  ( 619 ) -927-1297  DSN:  469-1297 

FAX:  (619)-939-2062 


MajGen  George  B  Harrison 

HQ  AFOTEC/CC 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117-5558 

OFF  TEL:  ( 505 ) -846-4533  DSN:  246-4533 

FAX:  (505)-846-9726 

E-mail ;  harrisonge@hq . af otec . af . mil 


Donald  L.  Gillett 
SAIC 

4242  Woodcock  Drive,  Suite  150 
San  Antonio  TX  78228 
OFF  TEL:  (210)-731-2220 
FAX:  (210)-731-2297 


Cheryl  Grant 

Illgen  Simulation  Technologies,  Inc 
Suite  10 
250  Storke  Road 
Goleta  CA  93117 
OFF  TEL:  (805 ) -562-3866 
FAX:  (805)-968-1311 


James  E  Haile 

AFMC  Office  of  Aerospace  Studies 

COEA  Support  Division 

3550  Aberdeen  Ave 

Kirtland  AFB  NM  87117-6008 

OFF  TEL:  {505)-846-8302  DSN:  246-8302 

FAX:  (505)-846-4668 


James  L.  Hanger  Jr. 

Aerospace  Corp 

The  Hallmark  Building,  Suite  187 
13873  Park  Center  Road 
Herndon  VA  22071 
OFF  TEL:  (703 )-318-1751 
FAX:  (703)-318-5409 


DR  Dean  S  Hartley  III 
Martin  Marietta  Energy  Systems 
Data  Systems  R&D  Program 
1099  Commerce  Park 
Oak  Ridge  TN  37830 
OFF  TEL:  ( 615 ) -574-7670 
FAX:  (615)-574-0792 
E-mail:  dhx@ornl.gov 
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Robert  G  Hart ling 

Chief  of  Naval  Operations  (N812D) 

2000  Navy  Pentagon 
Room  4A522 

Washington  DC  20350-2000 

OFF  TEL:  ( 703 ) -695-3797  DSN:  225-3797 

FAX:  (703)-693-9760 

E-mail:  rhart@dmso.dtic.dla.mil 


MSgt  Leon  F  Hass  - 

AFOTEC/OL-PC 

730  Irwin  Ave 

Falcon  AFS  CO  80912-7300 

OFF  TEL:  {719)-380-3238  DSN:  749-3238 

FAX;  (719) -380-3108 

E-mail ;  lhass@inet . ntf 1 . af • mil 


Capt  Phillip  A.  Herre 

31  TES  OL-KT 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117 

OFF  TEL:  ( 505 ) -846-5329  DSN:  246-5328 

FAX:  (505)-846-4285 


MAJ  Cyrus  E  Holliday 

TRADOC,  Threat  Support  Division 

Attn:  ATZL-CST 

Bldg  53,  700  Scott  Ave 

Fort  Leavenworth  KS  66027-1323 

OFF  TEL:  ( 913 ) -684-7974  DSN:  552-7974 

FAX:  (913)-684-2397 

E-mail :  hollidac@leal-emn.mil 


MAJ  Brian  Ishihara 
HQ  USAF/TEP 

1530  Air  Force  Pentagon 

Washington  DC  20330-1530 

OFF  TEL:  ( 703 ) -695-0900  DSN:  225-0900 

FAX:  (703)-695-0803 

E-mail :  ishiharb@tep .hq.af.mil 


DR  Kent  Haspert 
IDA 

1801  N.  Beauregard  Street 
Alexandria  VA  22311 
OFF  TEL:  ( 703 ) -845-2427 
FAX;  (703)-845-6911 


DR  Dale  B  Henderson 
Los  Alamos  National  Laboratory 
A  Division 
MS  F606 

Los  Alamos  NM  87545 
OFF  TEL:  ( 505 ) -665-2151 
FAX:  (505)-665-5204 
E-mail :  dbh@lanl . gov 


Stephen  Hobbs 
NCCOSC 
Code  784 

San  Diego  CA  92152-5001 
OFF  TEL;  ( 619 ) -553-2018 
FAX:  (619)-553-1635 
E-mail :  hobbs@nosc.mil 


Roger  K  Hoppe 
TASC 

2017  Yale  Avenue 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -764-8782 
FAX:  (505)-764-8754 


MAJ  Edward  B  Janeczko  Jr 

513th  Engineering  &  Test  Squadron 

901  SAC  Blvd  Suite  IHl 

Offutt  AFB  NE  68113-5520 

OFF  TEL:  (402 )-294-7027  DSN:  271-7027 

FAX:  (402)-294-6526 
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DR  Anil  N  Joglekar 
Institute  for  Defense  Analyses 
1801  Beauregard  St 
Alexandria  VA  22311 

OFF  TEL;  ( 703 ) -845-2343  DSN;  289-1825 
FAX;  (703)-845-6977 
E-mail :  ajoglekar@ida.org 


MAJ  Jeffrey  R  Kass 

OSD,  JADO/JEZ,  MOD/SIM  Branch 

307  West  Van  Matre  Ave 

Eglin  AFB  FL  32542-6805 

OFF  TEL:  ( 904 ) -882-5221  DSN:  872-5221 

FAX:  (904)-882-6215 


Carl  D  Keim 

BDM  International,  Inc 
1801  Randolph  Road,  SE 
Albuquerque  NM  87106 

OFF  TEL;  (505)-848-5368  DSN:  246-4682 
FAX:  (505)-846-1872 


James  C  Kolding 

Teledyne  Brown  Engineering 

300  Sparkman  Dr,  NW 

PO  Box  070007,  MS  #170 

Huntsville  AL  35807-7007 

OFF  TEL:  (205 ) -726-2893 

FAX:  {205)-726-2241 

E-mail:  jkolding@pobox.tbe.com 


Ann  Krause 

HQ  AFOTEC/SAN 

8500  Gibson  Blvd. ,  SE 

Kirtland  AFB  NM  87117-5558 

OFF  TEL:  ( 505 ) -846-2846  DSN:  246-2846 

FAX:  {505)-846-5145 

E-mail:  krausea@pl.afotec.af.mil 


Larry  J  Jurica 
The  MITRE  Corporation 
234  South  Fraly  Blvd 
Suite  100 
Dumfries  VA  22026 
OFF  TEL:  ( 703 ) -441-1775 
FAX:  (703)-441-1779 


Irwin  A  Kaufman 

Institute  for  Defense  Analyses 
1801  N.  Beauregard 
Alexandria  VA  22311 

OFF  TEL:  ( 703 ) -845-2486  DSN:  289-1868 
FAX:  (703)-845-6911 
E-mail;  ikaufman@ida.org 


Charles  Lee  Kirby 
US  Army  TRAC-WSMR 
ATTN:  ATARC-WJ 

White  Sands  Missile  Range  NM  88002-5502 
OFF  TEL;  ( 505 ) -678-1950  DSN:  258-1950 
FAX:  (505)-678-5104 
E-mail :  kirbyl@wsmr-emh91 .  army . mil 


LtCol  Kenneth  C.  Konwin 
JAST/PIA 

OFF  TEL;  ( 703 ) -602-7390  DSN:  332-7390 

Ext:  00000664 

FAX:  (703) -416-8440 

E-mail :  konwink@ntrprs  .jast.mil 


Philip  R  Landweer 
BDM  International 
MS  R-34 

1801  Randolph  Road,  SE 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -848-5251 
FAX :  ( 505 ) -848-4047 
E-mail:  plandwee@abq.bdm.com 
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Kenneth  E  Lavoie 

Air  Force  Wargaming  Center 

AUCADRE/WGT 

Bldg  1406 

Maxwell  APB  AL  36112-5532 

OFF  TEL;  (205) -953-6528  DSN;  493-6528 

FAX;  (205)-953-2593 

E-mail;  lavoie@cadre.af.mil 


Richard  R  Ledesma 

OSD,  OUSD(A&T)/DDT&E/A&SP 

The  Pentagon,  Room  3D1075 

Washington  DC  20301-3110 

OFF  TEL;  (703)-695-7245  DSN;  225-7245 

FAX;  (703)-614-7040 


DR  Gregory  A  Mack 
Booz  Allen  &  Hamilton 
Advanced  Decision  Systems  Div 


Daniel  McDonough 

AFOTEC/SAN 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117-5558 

OFF  TEL;  (505)-846-1359  DSN;  246-1359 

FAX;  (505) -846-5145 

E-mail ;  mcdonoudSpl . af otec . af.mil 


MAJ  Steven  A.  Minarik 

MARCORSYSCOM 

POB  156 

Quantico  VA  22134 

OFF  TEL;  (703) -640-3634  DSN;  278-3634 

FAX;  (703) -640-2168 

E-mail ;  minarik@psa@roarcorsyscom 


William  P.  Leavenworth 
SAIC 

c/o  OSD  JACO/JEZ 

307  W.  Van  Matre  Avenue,  Suite  105 
Eglin  AFB  FL  32542-6805 
OFF  TEL;  (904) -882-5221  DSN;  872-5221 
FAX;  (904)-882-6215 


COL  Gordon  A.  Long 
NORAD /USSPACECOM/AN 
Suite  116 

250  South  Peterson  Blvd 

Peterson  AFB  CO  80914-3180 

OFF  TEL;  ( 719) -554-3638  DSN;  692-3638 

FAX;  (719) -554-5068 

E-mail;  glong@spacecom.af.mil 


DR  Hugo  E  Mayer 

TRAC  Operations  Analysis  Center 

ATTN;  ATRC-FSD 

Fort  Leavenworth  KS  66027 

OFF  TEL;  (913) -684-6872  DSN;  552-6872 

FAX;  (913) -684-4368 

E-mail ;  mayerh@tracer . army . mil 


Robert  E  Mercer 

US  Army  Operational  Evaluation  Command 

4501  Ford  Avenue 

Alexandria  VA  22302-1458 

OFF  TEL;  (703)-756-1816  DSN;  289-1819 

FAX;  (703)-756-7586 


DR  Richard  B  Modjeski 
Operational  T&E  Command 
Park  Center  4 
4501  Ford  Ave 
Alexandria  VA  22302-1458 
OFF  TEL;  ( 703 ) -602-6038  DSN;  332-6039 
FAX;  (703)-602-6041 
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DR  Paul  R  Muessig 
Naval  Air  Warfare  Center 
Code  318 
Weapons  Div 

China  Lake  CA  93555-6001 

OFF  TEL:  ( 619 ) -927-1271  DSN:  469-1271 

FAX:  (619)-939-2062 


Patty  O’Brien 

HQ  AFOTEC/SAN 

8500  Gibson  Blvd. ,  SE 

Kirtland  AFB  NM  87117-5558 

OFF  TEL:  ( 505 ) -846-2845  DSN:  246-2845 

FAX:  (505)-846-5145 

E-mail :  obrienpSpl • af otec . af •mil 


DR  Julian  I  Palmore 

US  Army  Construct  Engineer  Rsch  Lab 

PO  Box  9005 

Champaign  IL  61826-9005 

OFF  TEL;  (217 ) -352-6511 

Ext:  00000768 

FAX;  (217)-373-6724 

E-mail :  palmore@osir is . cso . uiuc . edu 


DR  Albert  E.  Rainis 

OSD/OUSD(A&T)/TWP 

Pentagon 

Washington  DC  20330-3100 

OFF  TEL:  ( 703 ) -697-8183  DSN:  227-8183 

FAX:  (703)-614-7060 

E-mail:  rainisac@acg. osd.mil 


David  J  Reed 
IITRI 

185  Admiral  Cochrane  Drive 
Annapolis  MD  21401 
OFF  TEL:  ( 410) -573-7709 
FAX:  (410)-573-7760 
E-mail :  reedd© j  sc . mil 


Allen  M  Murashige 
AFSAA/SAT 

Scientific  &  Technical  Advisor 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  { 703 ) -697-5795  DSN:  227-5795 

FAX:  (703)-697-3441 

E-mail:  murashige@af saa. hq, af.mil 


Capt  Juan  C.  Ortiz 
31  TES,  OL-KT 
8500  Gibson  Blvd,  SE 
Kirtland  AFB  NM  87117-5558 
OFF  TEL:  ( 505 ) -846-5328  DSN:  246-5328 
FAX;  (505) -846-4285 
E-mail :  ort iz j @p2 . af otec .af.mil 


Steve  Payne 
DIA 

Missile  and  Space  Intel  Center 
Attn:  DIA/MSC-6B 

Redstone  Arsenal  AL  35898-5500 
OFF  TEL:  ( 205 ) -313-6231 
FAX:  {205)-313-6267 


Mark  H  Ralston 
US  Army  AMSAA 
Attn:  AMXSY-SL 

Aberdeen  Proving  Ground  MD  21005-5071 
OFF  TEL:  (410)-278-6577  DSN:  298-6577 
FAX:  (410)-278-2788 
E-mail :  ralston@arl.mil 


John  A  Riente 
HQ  Department  of  the  Army 
Deputy  Chief  of  Staff  for  Ops  &  Plans 
400  Army  Pentagon 
Washington  DC  20310-0400 
OFF  TEL:  ( 703 ) -697-4113  DSN:  227-4113 
FAX:  (703)-614-9044 
E-mail:  riente@pentemh2 .army .mil 
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John  R  Robbins 
Technical  Solutions  Inc. 
PO  box  1148 
Mesilla  Park  NM  88047 
OFF  TEL:  ( 505 ) -524-2154 
FAX:  (505) -525-5801 


Raybon  Sanderson 
BDM 

1801  Randolph  SE 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -848-5387 


DR  Ernest  A.  Seglie 
DOT&E/OSD 

Pentagon,  Room  3E318 

1700  Defense  Pentagon 

Washington  DC  20301-1700 

OFF  TEL:  ( 703 ) -697-7247  DSN:  227-7247 

FAX:  (703)-693-5248 

E-mail:  seglie-ernest@mail-host.dote.osd.mil 


DR  Patricia  A  Sanders 
0SD(PA&E) 

Land  Forces 

Pentagon,  Room  2B256 

Washington  DC  20301-1800 

OFF  TEL:  ( 703 ) -697-3521  DSN:  227-3521 

FAX:  (703)-693-5707 

E-mail:  pas@dmso.dtic.dla.mil 


MAJ  Joseph  R.  Schwab 
STRICOM 

Attn:  AMCPM-CATT 

12350  Research  Parkway 

Orlando  FL  32826 

OFF  TEL:  ( 407 ) -384-3226 

FAX:  (407) -384-3223 

E-mail:  schwabj@stricom.army.mil 


DR  John  C  Sessler 
BMDO/AQM 

The  Pentagon,  Room  1E149 
Washington  DC  20301 
OFF  TEL:  ( 703 ) -693-1594 
FAX:  (703)-693-1700 


LtCol  James  F  Sheedy 

HQ  AFOTEC/TF 

Kirtland  AFB  NM  87117 

OFF  TEL:  ( 505 ) -846-5326  DSN:  246-5326 

FAX:  (505) -846-5301 


Sheryl  L.  Shoupe 
BDM  International 
1801  Randolph  Road,  SE 
Albuquerque  NM  87106 
OFF  TEL:  ( 505 ) -848-5650 


LtCol  Robert  S  Sheldon 
AFSAA/SAZ 

The  Pentagon,  Room  1E386 
Washington  DC  20330-5420 
OFF  TEL:  (703 ) -693-8423  DSN:  227-8423 


James  J  Sikora 
BDM  International 
MS  BIO 

1801  Randolph  Road,  SE 

Albuc[uerque  NM  87106 

OFF  TEL:  ( 505 ) -848-5650 

FAX:  (505) -848-4051 

E-mail:  sikora@salsa.abq.bdm.com 
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Susan  D  Solick 

US  Army  TRAC-OAC 

Attn:  ATRC-FCA 

Fort  Leavenworth  KS  66027-5200 

OFF  TEL:  ( 913 ) -684-6868  DSN:  552-6868 

FAX:  (913)-684-4368 

E-mail:  solicks@tracer. army .mil 


DR  Walter  L  Stanley 

BDM  International 

1801  Randolph  Rd,  SE 

Albuquerque  NM  87106 

OFF  TEL:  ( 505 ) -848-5594 

FAX:  (505)-848-5528 

E-mail:  Stanley  @  abq.bdm.com 


Morris  A  Stringfield 
ATCOM,  Dept  of  the  Army 
AMSAT-D-BA 
4300  Goodfellow  Blvd 
St  Louis  MO  63120 

OFF  TEL:  (314) -263-3763  DSN:  693-3763 
FAX:  (314)-263-1148 


Clayton  J  Thomas  FS 
AFSAA/SAN 

1570  Air  Force  Pentagon 
Room  1E386 

Washington  DC  20330-1570 

OFF  TEL:  ( 703 ) -697-4300  DSN:  227-4300 

FAX:  (703)-697-3441 

E-mail :  thomasc@af saa . hq. af.mil 


Van  D  Underwood 

Johns  Hopkins  University/APL 

13550  Stowe  Drive 

Poway  CA  92064 

OFF  TEL:  ( 619 ) -486-5857 

FAX:  (619)-486-6201 

E-mail :  Underwood, VanD@cmpwest . edu 


Edward  N.  Sowell 
HQ  TEXCOM 

Headquarters  Avenue,  Bldg  91012 
Fort  Hood  TX  76544-5065 
OFF  TEL:  ( 817 ) -288-1845  DSN:  738-1845 
FAX:  (817)-288-1644 
E-mail:  txh2437@texcom-emhl.army.mil 


Capt  John  Stephenson 
HQ  AFOTEC/SAN 
8500  Gibson  Blvd.,  SE 
Kirtland  AFB  NM  87117-5558 
OFF  TEL:  ( 505 ) -846-7840  DSN:  246-7840 
FAX:  (505)-846-5145 
E-mail ;  stephan@pi . af otec . af . mil 


LTC  Howard  Taylor 

National  Simulation  Center 

Technical  Support  Director 

ATTN:  AT2L-NSC-T 

Fort  Leavenworth  KS  66027-7305 

OFF  TEL:  (913)-684-8184  DSN:  552-8184 

FAX:  (913)-684-8171 

E-mail:  taylorrl@leav-emh.army.mil 


LtCol  David  S  Thomen 

Marine  Corps  Combat  Development  Command 
Studies  and  Analysis  Division 
3093  Upshur  Ave 
Quantico  VA  22134-5130 
OFF  TEL:  ( 703 ) -640-3235  DSN:  278-3235 
FAX:  (703)-640-3547 
E-mail :  dthomen@dmso.dtic.dla.mil 


MAJ  Stephen  C  Upton 

Marine  Corps  Combat  Development  Command 
Studies  and  Analysis  Division 
3093  Upshur  Ave 
Quantico  VA  22134-5130 
OFF  TEL:  ( 703 ) -640-3235  DSN:  278-3235 
FAX:  (703)-640-3547 
E-mail :  uptons@mgg-smtpl . usmc . mil 
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Eugene  P  Visco  FS 
US  Army  MISMA 

Crystal  Square  2,  #808 >  SFUS“MIS 
1725  Jefferson  Davis  Hwy 
Arlington  VA  22202 

OFF  TEL:  (703)-607-3420  DSN:  327-3420 
FAX:  (703)-607-3381 

E-mail :  Eugene . P . Visco@Pentagon-ldmsl8 . army 
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Howard  G  Whitley  III 
US  Army  Concepts  Analysis  Agency 
8120  Woodmont  Ave 
Attn:  CSCA-TCT 

Bethesda  MD  20814-2797 
OFF  TEL:  (301)-295-1611  DSN:  295-1611 
FAX:  (301)-295-1287 
mil  E-mail:  whitley@caa.army.mil 


Richard  I  Wiles 

Military  Operations  Research  Society 
101  S  Whiting  Street 
Suite  202 

Alexandria  VA  22304 

OFF  TEL:  (703) -751-7290 

FAX:  (703)-751-8171 

E-mail:  rwiles@dgis.dtic.dla.mil 


Floyd  C.  Wofford 

US  Army  Materiel  Systems  Analysis  Acty 
ATTN:  AMXSY-CD 

Aberdeen  Proving  Ground  MD  21005-5071 
OFF  TEL:  (410)-278-5358  DSN:  298-5358 
FAX:  (410)-278-6585 


DR  Marion  L  Williams  FS 

HQ  AFOTEC  Chief  Scientist 

8500  Gibson  Blvd,  SE 

Kirtland  AFB  NM  87117-5558 

OFF  TEL:  (505) -846-0607  DSN:  246-0607 

FAX:  (505)-846-9726 

E-mail :  wi 1 1 iamsm@hq. af otec .af.mil 


DR  Robert  H  Wright 
Resource  Consultants,  Inc. 

3051  Technology  Pkwy,  #280 

Orlando  FL  32826 

OFF  TEL:  (407)-282-1451 

FAX:  (407)-658-9541 

E-mail:  wright@dmso.dtic.dla.mil 
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